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Abstract 



We present two sequences of ensembles of non-systematic irregular repeat-accumulate codes which asymp- 
totically (as their block length tends to infinity) achieve capacity on the binary erasure channel (BEC) with 
bounded complexity per information bit. This is in contrast to all previous constructions of capacity- achieving 
sequences of ensembles whose complexity grows at least like the log of the inverse of the gap (in rate) to 
capacity. The new bounded complexity result is achieved by puncturing bits, and allowing in this way a 
sufficient number of state nodes in the Tanner graph representing the codes. We also derive an information- 
theoretic lower bound on the decoding complexity of randomly punctured codes on graphs. The bound holds 
for every memory less binary- input output-symmetric channel and is refined for the BEC. 
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Index Terms: Binary erasure channel (BEC), codes on graphs, degree distribution (d.d.), density 
evolution (DE), irregular repeat-accumulate (IRA) codes, low-density parity-check (LDPC) codes, 
memoryless binary-input output-symmetric (MBIOS) channel, message-passing iterative (MPI) 
decoding, punctured bits, state nodes, Tanner graph. 
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1 Introduction 

During the last decade, there have been many exciting developments in the construction of low- 
complexity error-correction codes which closely approach the capacity of many standard communi- 
cation channels with feasible complexity. These codes are understood to be codes defined on graphs, 
together with the associated iterative decoding algorithms. By now, there is a large collection of 
these codes that approach the channel capacity quite closely with moderate complexity. 

The first capacity-achieving sequences of ensembles of low-density parity-check (LDPC) codes 
for the binary erasure channel (BEC) were found by Luby et al. [Z1IB] and Shokrollahi [H!]. Following 
these pioneering works, Oswald and Shokrollahi presented in a systematic study of capacity- 
achieving degree distributions (d.d.) for sequences of ensembles of LDPC codes whose transmission 
takes place over the BEC. Capacity-achieving ensembles of irregular repeat-accumulate (IRA) codes 

*The paper was presented in part at the 2004 IEEE International Symposium on Information Theory, Chicago, 
IL, USA, June 27-July 2, 2004. Parts of this paper will be also presented at the 23 rd IEEE Convention of Electrical 
& Electronics Engineers in Israel, September 2004, and at the 42th Annual Allerton Conference on Communication, 
Control and Computing, IL, USA, October 2004. The paper is submitted to the IEEE Trans, on Information Theory. 
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for the BEC were introduced and analyzed in [3J j an d also capacity-achieving ensembles for 
erasure channels with memory were designed and analyzed in jlOl 111] . 

In Khandekar and McEliece discussed the complexity of achieving the channel capacity on 
the BEC, and more general channels with vanishing bit error probability. They conjectured that 
if the achievable rate under message-passing iterative (MPI) decoding is a fraction 1 — e of the 
channel capacity, then for a wide class of channels, the encoding complexity scales like In - and the 
decoding complexity scales like - In ^. This conjecture is based on the assumption that the number 
of edges (per information bit) in the associated bipartite graph scales like ln^, and the required 
number of iterations under MPI decoding scales like -. However, for codes defined on graphs which 
are transmitted over a BEC, the decoding complexity under the MPI algorithm behaves like In - 
(same as encoding complexity) El EE] • This is since the absolute reliability provided by the 
BEC allows every edge in the graph to be used only once during MPI decoding. 

In Jl] , Sason and Urbanke considered the question of how sparse can parity-check matrices of 
binary linear codes be, as a function of their gap (in rate) to capacity (where this gap depends 
on the channel and the decoding algorithm). If the code is represented by a standard Tanner 
graph without state nodes, the decoding complexity under MPI decoding is strongly linked to 
the density of the corresponding parity-check matrix (i.e., the number of edges in the graph per 
information bit). In particular, they considered an arbitrary sequence of binary linear codes which 
achieves a fraction 1 — e of the capacity of a memoryless binary-input output-symmetric (MBIOS) 
channel with vanishing bit error probability. By information-theoretic tools, they proved that for 
every such sequence of codes and every sequence of parity-check matrices which represent these 

codes, the asymptotic density of the parity-check matrices grows at least like Kl+ ^J n e where 
K\ and K<i are constants which were given explicitly as a function of the channel statistics (see 
|14l Theorem 2.1]). It is important to mention that this bound is valid under ML decoding, and 
hence, it also holds for every sub-optimal decoding algorithm. The tightness of the lower bound 
for MPI decoding on the BEC was demonstrated in |14[ Theorem 2.3] by analyzing the capacity- 
achieving sequence of check-regular LDPC-code ensembles introduced by Shokrollahi |16j . Based 
on the discussion in (14] . it follows that for every iterative decoder which is based on the standard 
Tanner graph, there exists a fundamental tradeoff between performance and complexity, and the 
complexity (per information bit) becomes unbounded when the gap between the achievable rate and 
the channel capacity vanishes. Therefore, it was suggested in ^3] to study if better performance 
versus complexity tradeoffs can be achieved by allowing more complicated graphical models (e.g., 
graphs which also involve state nodes). 

In this paper, we present sequences of capacity-achieving ensembles for the BEC with bounded 
complexity under MPI decoding. The new ensembles are non-systematic IRA codes with properly 
chosen d.d. (for background on IRA codes, see [I] and Section |2J). The new bounded complexity 
results improve on the results in |15j . and demonstrate the superiority of properly designed non- 
systematic IRA codes over systematic IRA codes (since with probability 1, the complexity of any 
sequence of ensembles of systematic IRA codes becomes unbounded under MPI decoding when the 
gap between the achievable rate and the capacity vanishes |15| Theorem 1]). The new bounded 
complexity result is achieved by allowing a sufficient number of state nodes in the Tanner graph 
representing the codes. Hence, it answers in the affirmative a fundamental question which was 
posed in ^1] regarding the impact of state nodes in the graph on the performance versus complexity 
tradeoff under MPI decoding. We suggest a particular sequence of capacity-achieving ensembles of 
non-systematic IRA codes where the degree of the parity-check nodes is 5, so the complexity per 
information bit under MPI decoding is equal to when the gap (in rate) to capacity vanishes (p 
designates the bit erasure probability of the BEC). Computer simulation results for these ensembles 
appear to agree with this analytical result. It is worth noting that our method of truncating the 
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check d.d. is similar to the bi-regular check d.d. introduced in ^H] for non-systematic IRA codes. 

We also derive in this paper an information-theoretic lower bound on the decoding complexity of 
randomly punctured codes on graphs. The bound holds for every MBIOS channel with a refinement 
for the particular case of a BEC. 

The structure of the paper is as follows: Section |2 provides preliminary material on ensembles 
of IRA codes, Section El presents our main results which are proved in Section |3J Analytical and 
numerical results for the considered degree distributions and their asymptotic behavior are discussed 
in Section [5J Practical considerations and simulation results for our ensembles of IRA codes are 
presented in Section H3 We conclude our discussion in Section Three appendices also present 
important mathematical details which are related to Sections |I] and 03 



2 IRA Codes 

We consider here ensembles of non-systematic IRA codes. We assume that all information bits 
are punctured. The Tanner graph of these codes is shown in Fig. ^ These codes can be viewed 
as serially concatenated codes, where the outer code is a mixture of repetition codes of varying 
order and the inner code is generated by a differential encoder with puncturing. We define these 
ensembles by a uniform choice of the interleaver separating the component codes. 




Figure 1: The Tanner graph of IRA codes. 

Using standard notation, an ensemble of IRA codes is characterized by its block length n and 
its d.d. pair A(x) = YliLi Kx 1 ' 1 and p{x) = Y^L\Pi x ■ Here, A, (or p,, respectively) designates 
the probability that a randomly chosen edge (among the edges that connect the information nodes 
and the parity-check nodes) is connected to an information bit node (or to a parity-check node) of 
degree i. As is shown in Fig. Q every parity-check node is also connected to two code bits; this 
is a result of the differential encoder which is the inner code of these serially concatenated and 
interleaved codes. Let R(x) = ^4 x% be a power series where the coefficient Ri denotes the 

fraction of parity-check nodes that are connected to i information nodes. Therefore, R(-) and p(-) 
are related by the equation 

We assume that the random permutation in Fig. ^ is chosen with equal probability from the 
set of all permutations. A randomly selected code from this ensemble is used to communicate over 
a BEC with erasure probability p. The asymptotic performance of the MPI decoder (as the block 
length tends to infinity) can be analyzed by tracking the average fraction of erasure messages which 
are passed in the graph of Fig-^during the I th iteration. This technique was introduced in ^2] and 
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is known as density evolution (DE). In the asymptotic case where the block length tends to infinity, 
the messages which are passed through the edges of the Tanner graph are statistically independent. 

Using the same notation as in [I], let Xq be the probability of erasure for a message from 
information nodes to parity-check nodes, x± be the probability of erasure from parity-check nodes 
to code nodes, x^p be the probability of erasure from code nodes to parity-check nodes, and finally, 
let Xg be the probability of erasure for messages from parity-check nodes to information nodes (see 
Fig. We now assume that we are at a fixed point of the MPI decoding algorithm, and solve for 
xo- We obtain the following equations: 

x x = 1 - (1 - x 2 )R(l - x ), (2) 
x 2 =px 1 , (3) 
X3 = l-(l-x 2 ) 2 />(i-zo), (4) 

x = A(x 3 ). (5) 

The only difference between equations (2)— (5) and those in [3] is the absence of a p in (5). This 
modification stems from the fact that all information bits are punctured in the ensemble considered. 
Solving this set of equations for a fixed point of iterative decoding provides the equation 



x = A 1 



1 n 2 

1-p 



l-pfl(l-xo) 



p(l - x ) . (6) 



If Eq. Q has no solution in the interval (0, 1], then according to the DE analysis of MPI decoding, 
the bit erasure probability must converge to zero. Therefore, the condition that 



A 1 



i 1 2 

1 — p 



l-pR(l - x) 



p(l-x) < x, Vx G (0,1], (7) 



implies that MPI decoding obtains a vanishing bit erasure probability as the block length tends to 
infinity. 

The design rate of the ensemble of non-systematic IRA codes can be computed by matching 
edges in the Tanner graph shown in Fig. ^ In particular, the number of edges in the permutation 
must be equal to both the number of information bits times the average information bit degree 
and the number of code bits times the average parity-check degree (ignoring the parity-check edges 
connected to the differential encoder). This implies that the design rate is equal to 

^ira = Jo A(x) dx 
Jo P( x ) dx 

Furthermore, we will see that R l ^ A = 1 — p for any pair of d.d. (A, p) which satisfies Eq. © for all 
xo 6 [0,1]. 

In order to find a capacity-achieving ensemble of IRA codes, we generally start by finding a 
d.d. pair (A, p) with non-negative power series expansions which satisfies Eq. © for all xo G [0, 1]. 
Next, we slightly modify A(-) or p(-) so that Eq. (JJJ) is satisfied and the new design rate in Eq. (jSJ 
is equal to (1 — e)(l —p) for an arbitrarily small e > 0. Since the capacity of the BEC is 1 — p, this 
gives an ensemble which has vanishing bit erasure probability under MPI decoding at rates which 
are arbitrarily close to capacity. 
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3 Main Results 



Definition 1. Let {C m } be a sequence of binary linear codes of rate R m , and assume that for 
every m, the codewords of the code C m are transmitted with equal probability over a channel 
whose capacity is C. This sequence is said to achieve a fraction 1 — e of the channel capacity with 
vanishing bit error probability if lmim^oo R m > (1 — s)C, and there exists a decoding algorithm 
under which the average bit error probability of the code C m tends to zero in the limit where m 
tends to infinity. 1 

Definition 2. Let C be an ensemble of LDPC or IRA codes whose d.d., A(-) and p(-), can be chosen 
arbitrarily (subject to possibly some constraints). The encoding and the decoding complexity are 
measured in operations per information bit which are required for achieving a fraction 1 — e of the 
channel capacity with vanishing bit erasure probability. Unless the pair of d.d. is specified, the 
encoding and the decoding complexity are measured with respect to the best ensemble (i.e., for the 
optimized pair of d.d.), and refer to the average complexity over this ensemble (as the block length 
of the codes tends to infinity, the complexity of a typical code from this ensemble concentrates to 
the average complexity). We denote the encoding and the decoding complexity by xe(z,C) and 
Xd(£,C), respectively. 



We note that for the BEC, both the encoding and the decoding complexity of IRA codes under 
MPI decoding is equal to the normalized number of edges per information bit in the associated 
Tanner graph. 

Theorem 1 (Bit-Regular Ensembles with Bounded Complexity). Consider the ensemble 
of bit-regular non-systematic IRA codes C, where the d.d. of the information bits is given by 

X(x)=x q - 1 , q>3 (9) 

which implies that each information bit is repeated q times. Assume that the transmission takes 
place over a BEC with erasure probability p, and let the d.d. of the parity-check nodes 2 be 

/>(*) = 7 ; ^P-)* _-. (10) 



1 — p 1 1 — qx + (q — 1) 



1 - (1 - X) 9-1 



Let p n be the coefficient of x n 1 in the power series expansion of p(x) and, for an arbitrary e 6 (0, 1), 
define M(e) to be the smallest positive integer 3 M such that 

oo 

n=M+l Hy 11 

The e-truncated d.d. of the parity-check nodes is given by 

/ M(e) \ M(e) 

p £ {x) = 1 - J2 Pn + J2 ■ ( 12 ) 

\ n=2 / n=2 



1 We refer to vanishing bit erasure probability for the particular case of a BEC. 

2 The d.d. of the parity-check nodes refers only to the connection of the parity-check nodes with the information 
nodes. Every parity-check node is also connected to two code bits (see Fig. 0, but this is not included in p(x). 

3 The existence of M{e) for e £ (0, 1) follows from the fact that p n = 0{n~ q ^ q ~ 1 ^). This implies that X1^a/+i P" 
can be made arbitrarily small by increasing M. 



5 



For q = 3 and p G (0, jL], the polynomial /%(•) has only non-negative coefficients, and the d.d. pair 
(A, p £ ) achieves a fraction 1 — e of the channel capacity with vanishing bit erasure probability under 
MPI decoding. Moreover, the complexity (per information bit) of encoding and decoding satisfies 

XE {e,C)= X D{e,C) <q + - r- -. (13) 

(1 - e) 

In the limit where e tends to zero, the capacity is achieved with a bounded complexity of q + 

Theorem 2 (Check- Regular Ensemble with Bounded Complexity). Consider the ensemble 
of check-regular non-systematic IRA codes C, where the d.d. of the parity-check nodes is given by 

p(x) = x 2 . (14) 

Assume that the transmission takes place over a BEC with erasure probability p, and let the d.d. 
of the information bit nodes be 4 



A(*) = l + ^ ^ , LL. (15) 



2p(l — ./• )"' sin ( A arcsin ' • lp> J - : 



2 



(i-pF 

Let A n be the coefficient of x n_1 in the power series expansion of X(x) and, for an arbitrary e G (0, 1) 
define M(e) to be the smallest positive integer 5 M such that 



n=A/+l 



This infinite bit d.d. is truncated by treating all information bits with degree greater than M{e) as 
pilot bits (i.e., these information bits are set to zero). Let X e ( x ) be the e-truncated d.d. of the bit 
nodes. Then, for all p G [0,0.95], the polynomial A e (-) has only non-negative coefficients, and the 
modified d.d. pair (A e , p) achieves a fraction 1 — e of the channel capacity with vanishing bit erasure 
probability under MPI decoding. Moreover, the complexity (per information bit) of encoding and 
decoding is bounded and satisfies 

XE (£,C) = xd(e,C) < (1 _ p) 5 (1 _ g) - (17) 

In the limit as e tends to zero, the capacity is achieved with a bounded complexity of j- 5 — . 



The following two conjectures extend Theorems ^ an d to a wider range of parameters. Both 
of these conjectures can be proved by showing that the power series expansions of \(x) and p{x) 
are non-negative for this wider range. Currently, we can show that the power series expansions of 
\(x) and p{x) are non-negative over this wider range only for small values of n (using numerical 
methods) and large values of n (using asymptotic expansions). We note that if these conjectures 
hold, then Theorem^is extended to the range p G [0, ^] (as q — > oo), and Theorem 121 is extended 
to the entire range p G [0, 1). 

4 For real numbers, one can simplify the expression of X(x) in However, since we consider later A(-) as a 

function of a complex argument, we prefer to leave it in the form of l|15|l . 

5 The existence of M(e) for e G (0, 1) follows from the fact that A n = 0(n~ 3 ' 2 ). This implies that ^nLjif+i ^ n / n 
can be made arbitrarily small by increasing M. 
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Conjecture 1. The result of Theorem ^ also holds for q > 4 if 

6 - 7q + 2q 2 

P < < 



4 < q < 



+ W . (18) 



12-17<? + 6g^ 

q>9 



12 - 37q + 26q 2 



We note that the form of Eq. (|18j) is implied by the analysis in Appendix A. 
Conjecture 2. The result of Theorem El also holds for p G (0.95, 1). 

In continuation to Theorem |^1 and Conjecture [21 it is worth noting that Appendix C suggests 
a conceptual proof which in general could enable one to verify the non-negativity of the d.d. 
coefficients {A n } for p 6 [0, 1 — e], where e > can be made arbitrarily small. This proof requires 
though to verify the positivity of a fixed number of the d.d. coefficients, where this number grows 
considerably as e tends to zero. We chose to verify it for all n G N and p £ [0,0.95]. We note 
that a direct numerical calculation of {A n } for small to moderate values of n, and the asymptotic 
behavior of A n (which is derived in Appendix B) strongly supports Conjecture |2l 

Theorem 3 (Information-Theoretic Bound on the Complexity of Punctured Codes over 
the BEC). Let {C' m } be a sequence of binary linear block codes, and let {C m } be a sequence of 
codes which is constructed by randomly puncturing information bits from the codes in {C' m }. e Let 
-fpct designate the puncturing rate of the information bits, and suppose that the communication 
of the punctured codes takes place over a BEC with erasure probability p, and that the sequence 
{C m } achieves a fraction 1 — e of the channel capacity with vanishing bit erasure probability. Then 
with probability 1 w.r.t. the random puncturing patterns, and for an arbitrary representation of 
the sequence of codes {C' m } by Tanner graphs, the asymptotic decoding complexity under MPI 
decoding satisfies 

I ln f— ) \ 
liminf XD(C m ) > } " \ + U (19) 

1_P W(l=fe) / 

where 

Peff = l-(1-Ppct)(l-P) (20) 

and £ m i n designates the minimum number of edges which connect a parity-check node with the 
nodes of the parity bits. 7 Hence, a necessary condition for a sequence of randomly punctured codes 
{C m } to achieve the capacity of the BEC with bounded complexity is that the puncturing rate of 
the information bits satisfies the condition P pct = 1 — 0(e). 

Theorem |1] suggests an extension of Theorem El though as will be clarified later, the lower 
bound in Theorem |3] is at least twice larger than the lower bound in Theorem |1] when applied to 
the BEC. 



6 Since we do not require that the sequence of original codes {C' m } is represented in a systematic form, then by 
saying 'information bits', we just refer to any set of bits in the code C' m whose size is equal to the dimension of 
the code and whose corresponding columns in the parity-check matrix are linearly independent. If the sequence of 
the original codes {C'm} is systematic (e.g., turbo or IRA codes before puncturing), then it is natural to define the 
information bits as the systematic bits of the code. 

7 The fact that the value of l m in can be changed according to the choice of the information bits is a consequence 
of the bounding technique. 
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Theorem 4 (Information-Theoretic Bound on the Complexity of Punctured Codes: 
General Case). Let {C' m } be a sequence of binary linear block codes, and let {C m } be a sequence 
of codes which is constructed by randomly puncturing information bits from the codes in {C^l- Let 
-Fpct designate the puncturing rate of the information bits, and suppose that the communication 
takes place over an MBIOS channel whose capacity is equal to C bits per channel use. Assume 
that the sequence of punctured codes {C m } achieves a fraction 1 — e of the channel capacity with 
vanishing bit error probability. Then with probability 1 w.r.t. the random puncturing patterns, 
and for an arbitrary representation of the sequence of codes {C' m } by Tanner graphs, the asymptotic 
decoding complexity per iteration under MPI decoding satisfies 



and f(y) = p(y\x = 1) designates the conditional pdf of the channel, given the input is x = 1. 
Hence, a necessary condition for a sequence of randomly punctured codes {C m } to achieve the 
capacity of an MBIOS channel with bounded complexity per iteration under MPI decoding is that 
the puncturing rate of the information bits satisfies P pc t = 1 — 0(e). 

Remark 1 (Deterministic Puncturing). It is worth noting that Theorems |3] and 0] both depend 
on the assumption that the set of information bits to be punctured is chosen randomly. It is an 
interesting open problem to derive information-theoretic bounds that apply to every puncturing 
pattern (including the best carefully designed puncturing pattern for a particular code). We also 
note that for any deterministic puncturing pattern which causes each parity-check to involve at 
least one punctured bit, the bounding technique which is used in the proofs of Theorems |3] and 
becomes trivial and does not provide a meaningful lower bound on the complexity in terms of the 
gap (in rate) to capacity. 




(21) 



where 




(22) 
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4 Proof of the Main Theorems 

In this section, we prove our main theorems. The first two theorems are similar and both prove 
that under MPI decoding, specific sequences of ensembles of non-systematic IRA codes achieve 
the capacity of the BEC with bounded complexity (per information bit). The last two theorems 
provide an information-theoretic lower bound on the decoding complexity of randomly punctured 
codes on graphs. The bound holds for every MBIOS channel and is refined for the particular case 
of a BEC. 

The approach used in the first two theorems was pioneered in [7] and can be broken into roughly 
three steps. The first step is to find a (possibly parameterized) d.d. pair (A, p) which satisfies the 
DE equation ©. The second step involves constructing an infinite set of parameterized (e.g., 
truncated or perturbed) d.d. pairs which satisfy the DE inequality (0. The third step is to verify 
that all of coefficients of the d.d. pair (A, p) are non-negative and sum to one for the parameter 
values of interest. Finally, if the design rate of the ensemble approaches 1 — p for some limit point 
of the parameter set, then the ensemble achieves the channel capacity with vanishing bit erasure 
probability. The following lemma simplifies the proof of Theorems ^ and EJ We note that its proof 
is based on the analysis of capacity-achieving sequences for the BEC in dfT and the extension to 
erasure channels with memory in |lfll 111 j . 

Lemma 1. Any pair of d.d. functions (A,p) which satisfy A(0) = 0, A(l) = 1, and satisfy the 
DE equation © for all xq S [0, 1] also have a design rate (jHJ) of 1 — p (i.e., it achieves the channel 
capacity of a BEC whose erasure probability is p). 

Proof. We start with Eq. @ and proceed by substituting xq = 1 — x, applying A -1 (-) to both sides, 
and moving things around to get 



Since A(-) is positive, monotonic increasing and A(0) = 0, A(l) = 1, we can use the identity 




(23) 



Integrating both sides from x = to x = 1 gives 





(24) 



to show that 




Taking the derivative of both sides of Eq. 



(^Q) shows that 




and then it follows easily that 
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where the fact that R(0) = and -R(l) = 1 is implied by Eq. (fTJl. Dividing both sides by the 
integral of p(-) and using Eq. (jBJ) shows that the design rate i? IRA = 1 — p. □ 

4.1 Proof of Theorem [T] 
4.1.1 Finding the D.D. Pair 

Consider a bit-regular ensemble of non-systematic IRA codes whose d.d. pair (A, p) satisfies the 
DE equation Q. We approach the problem of finding the d.d. pair by solving Eq. (jHJ) for p(-) in 
terms of A(-) and the progression is actually similar to the proof of Lemma ^ except that the limits 
of integration change. Starting with Eq. (|23ft and integrating both sides from x = to x = t gives 

*> l-p YR>( x ) dx 

(25) 



o y l-pR(x)J R'(l) 
R'(l) l-pR{ty 



where the substitution p{x) = R'(x)/R'(l) follows from Eq. The free parameter R'(l) can be 
determined by requiring that the d.d. R(-) satisfy R(l) = 1. Solving Eq. (|25|) for R'(l) with t = 1 
and R(l) = 1 shows that 

R'(l) = ^ . (26) 

£(l-\-i{l-x)) dx 

Solving Eq. l(23|) for R(t) and substituting for -R'(l) gives 

;*(i-a-hi-*)) ^ 
R ,q = ^ (l-A-iq-g))^ 



fi(l-X-i(l- x ))dx 



For simplicity, we now define 



substitute x for t, and get 



1 -p + pQ(x 

It follows from Eqs. (Q) and (JSEJ) that 

R'(x) 



Q{x) = — - — > 27 

£ (1 - A-i(l - t» dt' 



R(x) = Qi - X \ (28) 



p{x) 



R>(1) 

(l-p)Q'(x) (l-p + p Q(l)f 
{l-p + P Q(x)f (l-p)Q'(l) 

1 Q'(x) 
(l-p + pQ{x)f Q'(l) 
\-\-\l-x) 
(l-p + pQix)) 21 



(29) 
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and 

_ !-*-«» - _ !. (30) 

(i-P + ?Q(l)) 2 

The important part of this result is that there is no need to truncate the power series of p(-) to 
force p(l) = 1. This appears to be an important element of ensembles with bounded complexity. 

Now, consider the bit-regular case where every information bit is repeated q > 3 times (i.e., 
A(x) = x q ~ l ). From Eq. (23), it can be verified with some algebra that 

Q(x) = qx - (q - 1) 1 - (1 - ar)A . (31) 

Substituting this into Eq. (|29(1 gives the result in Eq. 1)10(1. 



Finally, we show that the power series expansion of Eq. (|10j) defines a proper probability dis- 
tribution. First, we note that p G (0, y^] by hypothesis, and that Appendix A establishes the 
non-negativity of the d.d. coefficients {p n } under this same condition. Since p(l) = 1, these coef- 
ficients must sum to one if the power series expansion converges at x = 1. This follows from the 
asymptotic expansion, given later in (|63j) . which implies that p n = 0(n~ q /( q ~ 1 ^). Therefore, the 
function p(x) defines a proper d.d. 



4.1.2 Truncating the D.D. 

Starting with the d.d. pair (A,p) implied by Eq. (|29|) (which yields that Eq. © holds), we apply 
Lemma n t° show that the design rate is 1 — p. The next step is to slightly modify the check d.d. 
so that the inequality is satisfied instead. In particular, one can modify the p(x) from (|29|) so 
that the resulting ensemble of bit-regular non-systematic IRA codes is equal to a fraction 1 — s of 
the BEC capacity. 

Let us define M(e) to be the smallest positive integer M such that the condition in 1)11(1 is 
satisfied. Such an M exists for any e S (0, 1) because p n = 0{n~ q ^ q ~ 1 ^). We define the e-truncation 
of p(-) to be the new check degree polynomial in ()12)1 . which is also equal to 

/ oo \ M(e) 

p £ ( X ) =\ P x+ pA + Y. ( 32 ) 

\ i=M(e)+l / i=2 

and satisfies p e (l) = p(l) = 1. Based on Eqs. (fTT|) and (|3*2"|) . and the fact that the power series 
expansion is non-negative, it is easy to show that 



/ Pe(x) dx < p(x) dx + 
Jo Jo 



1 + e 



q{l-p) q{l-p)' 



Applying Eq. (jHJ) to the last equation, shows that the design rate of the new ensemble (A, p e ) of 
bit-regular, non-systematic IRA codes is given by 

^ira = Jo X ( x ) dx = 1 > !~P 

Jo p £ {x)dx q Jq p e {x)dx 1 + e 

Using the fact that > 1 — e, for e > 0, we get the final lower bound 

R IRA > (l-p)(l-e). (33) 
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This shows that the design rate of the new ensemble of codes is equal at least to a fraction 1 — e of 
the capacity of the BEC. Now, we need to show that the new ensemble satisfies the inequality (JJJ), 
which is required for successful decoding, given by 



A 1 



1—p 



l- P R £ (l-x) 



p £ (l-x) <x, ViG (0,1] (34) 



where R £ (-) can be computed from p £ (-) via Eq. (^Q). Since the truncation of p(x) only moves edges 
from high degree checks (i.e., x J terms with j > M) to degree one checks (i.e. the x° term), it 
follows that 

Pe (x)>p(x), x€[0,l). (35) 

We will also show that 
Lemma 2. 

R £ (x)>R(x), a; €(0,1). (36) 
Proof. We rely on Eqs. (fTTTft and (|32|) to show that for an arbitrary e > 



Pi + YaLm{e)+i Pi , E 



M(e) p 



■i ~a 



oo 



R £ (x) = K1 ' ^ me)+1 " n ■ x + 2 r " .... ± V (37) 

Pi + 2-,i=Af(e)+l P* 2^j=2 T P 1 ~r 2^i=M{e)+l Pi "•" 2^i=2 T t=l 



rM(e) p. 

PI t Z-,i=Af(e)+l ^ T" Z^i=2 T ^ T Z-/i=M(e)+l Pi Z. 

and 



i2(x) = ^ r^r^- -x + ^= 2 4 " X ,„ , ^YR iX \ (38) 

Pi "+" Zvt=Af(e)+l » 2^j=2 j Pi 2^i=M(e)+l j "+" 2^j=2 j i=l 

It is easy to verify that the coefficients in the power series expansions of R £ (-) and R(-) in 1)3 7 Jl 
and (|38jl. respectively, are all non-negative and each of them sum to one. By comparing the two, 
it follows easily that 



R\ £) <Ri V i > 2 . 



Since Yli^i Ri = Yli^i R-i = 1, we also see that 



Let 



R^ ^ > R\. 



6i±Ri-l6 e) i = 1,2,... 



then Si = — X^2 $i (si nce by definition, YliLi &i = z3£i ^ ~~ z3£i = 0)> an< ^ $i > f° r every 
integer i > 2. It therefore follows that for x € (0, 1) 

00 00 
R(x) - R £ {x) = Six + ^2 $i x% < ^ + ^2 = 

i=2 i=2 

which proves the inequality in (|3fijl. □ 

Finally, the validity of the condition in l|34j) follows immediately from the two inequalities in 
(|3*5|) and (|3f))) . and the fact that the d.d. pair (A, p) satisfies the equality in © for all xq £ [0, 1]. 
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4.2 Proof of Theorem [U 



4.2.1 Finding the D.D. Pair 

Like in the proof of Theorem Q we start the analysis by solving equation (|25|1. but this time we 
calculate A(-) for a particular choice of p(-). Let us choose p(x) = x 2 , so R(x) = x 3 , and we obtain 
from (j23j) that the inverse function of A(-) is equal to 

A "w = 1 -( i-rti-^ ) 2(1 -* )2 - (39> 

Inserting (|39ft into ()15|) shows that the expression of A(-) in (O is the inverse function to (j39j) 
for x G [0, 1], so ((T3|) gives us a closed form expression of A(-) in the interval [0, 1]. As we noted 
already, for real numbers, one can simplify the expression of A(-) in (|15|) . but since we consider it 
later as a function of a complex argument, then we prefer to leave it in the form of ()15|) , 

In the following, we show how (|15|) was derived. Note that since we already verified the correct- 
ness of (|15[). then in the following derivation we do not need to worry about issues of convergence. 
Set 

y = A tx) , z = . u = 1 — x, 

1 — p 

Then with these notation and for u £ [0, 1], (j39j) can be written in the form 

z<f>(u) = u 

where 4>{u) = 1 — pv?. We now use the Lagrange inversion formula (see, e.g., ^ Section 2.2]) to 
obtain the power series expansion of u = u{z) around z = 0, i.e., we write 

oo 

u{z) = ^U k Z k . 
k=0 

If z = then u = 0, so no = u(0) = 0. The Lagrange inversion formula states that 

u k = \ \u k - x \^{v), fc = l,2,... (40) 

where [u k ~ l ](j) k (u) is the coefficient of u k ~ 1 in the power series expansion of (j) k {u). From the 
definition of </>(•), the binomial formula gives 

^{u) = (i- P u 3 ) k = {(-i) j Qy^j (4i) 



j=0 



so 



from (gnj) and (gTJ) 



We conclude that 



, t V^(A)p^, if k = 1, 4, 7, 10, 
otherwise 



fc: ^ieN I V 3 7 
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where N designates the set of non-negative integer numbers. From z = ^i_p , we get 



(i-y) 



1 



2 



(l-p) k (l~p) k 

so the composition of the two equalities above gives 

(-1) 3 / k \ p 



n=0 



E 



A: 



fc- 1 oo 



and x = 1 — u = X(y) (since y = A (x)). Finally, we obtain a power series expansion for A(-) from 
the last two equalities 



A(x) = l- J2 



(-1)— 



k: 



k \ k ~s 1 °° 

*f0 (T^F ' 5> i; 



n=0 



n I 2 \ x n 



By substituting = 3/ + 1 where / € N, the latter equation can be written as 



X(x) 



1 fJ(-l) 1 
1 y [(-!)' 



i ) \i- P 



31+1 oo /31+1 



E 

n=0 



J? 



(-*)» 



3/+1 



(1 - XJ 2 



L » r_(-iy /3z + i\ 3l+1 \ 

ft f^l3/ + l V I J J 



where t = \/l ~~ x - Fortunately, the final sum can be expressed in closed form and leads to 

the expression of A(-) in (|15[). Plots of the function A(-) as a function of p £ (0, 1) are depicted in 
Fig. El 




Figure 2: The function A(-) in (|T3|) . as a function of the erasure probability p of the BEC. 
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Finally, we show that the power series expansion of Eq. (|15|) defines a proper probability distri- 
bution. Three different representations of the d.d. coefficients {A n } are presented in Section [5.2.11 
and derived in Appendix B. They are also used in Appendix C to prove the non- negativity of the 
d.d. for p G [0,0.95]. Since A(l) = 1, these coefficients must also sum to one if the power series 
expansion converges at x = 1. The fact that A n = 0(n -3 / 2 ) follows from a later discussion (in 
Section [5.2.2(1 and establishes the power series convergence at x = 1. Therefore, the function X(x) 
gives a well-defined d.d. 



4.2.2 Truncating the D.D. 

Now, we must truncate A(-) in such a way that inequality 0, which is a necessary condition for 
successful iterative decoding, is satisfied. We do this by treating all information bits with degree 
greater than some threshold as pilot bits. In practice, this means that the encoder uses a fixed 
value for each of these bits (usually zero) and the decoder has prior knowledge of these fixed values. 
This truncation works well because a large number of edges in the decoding graph are initialized by 
each pilot bit. Since bits chosen to be pilots no longer carry information, the cost of this approach 
is a reduction in code rate. The rate after truncation is given by 

R iKA = K^ = KK^ = Kf K-K> 



N N K N \ K 

where N is the block length, K is number of information bits before truncation, and K' is the 
number of information bits after truncation. Applying Lemma^to the d.d. pair (A,p) shows that 
the design rate is given by K/N = 1—p. Therefore, the rate can be rewritten as i? IRA = (1— p)(l — 5) 
where S = (K — K')/ K is the fraction of information bits that are used as pilot bits. 

For an arbitrary e E (0, 1), we define M{e) to be the smallest positive integer M which satisfies 
Eq. 1)16(1 . Next, we choose all information bit nodes with degree greater than M(e) to be pilot 
bits. This implies that the fraction of information bit nodes used as pilot bits is given by 8 = 
Y^=M{e)+\ Ln where the fraction of information bit nodes with degree n is given by 



Ln = v i n/ ; , ■ (42) 

En=2 X n/n 



Based on Eqs. (jHJ) and (fTI() . we have 



Y^l= \(x)dx = R mA / p( x )dx = ^. (43) 

n Jo Jo 3 



n 

n=2 



Therefore, we can use Eqs. ()T7)() . P^)) and ((IH)) to show that 

x _ ^ _ X^M(e)+l W n _ S^La/(£)+1 "W n 

o — 2 . L n - 



< 



(l-p)g 

3 



n=M(e)+l Ln=2*n/n — — 



Let us define the effective e-modified d.d. to be 



M(s) 

A £ (x) = ^ A n 



x n ~ l 



n=2 



Although this is not a d.d. in the strict sense (because it no longer sums to one), it is the correct A 
function for the DE equation. This is because all information bits with degree greater than M{e) 
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are known at the receiver and therefore have zero erasure probability. Since the d.d. pair (X,p) 
satisfies the equality in (j^J) and X £ (x) < A(x) for x £ (0,1], then it follows that the inequality 
required for successful decoding (J7|) is satisfied. 

As explained in Section ^ the encoding and decoding complexity on the BEC are both equal 
to the number of edges, per information bit, in the Tanner graph. The degree of the parity-check 
nodes is fixed to 5 (three edges attached to information bits and two edges attached to code bits), 
and this implies that the complexity is given by 

XE (e,C) = Xn(s,C) = < {1 _ p " {1 _ £ y 
Therefore, the complexity is bounded and equals j^— as the gap to capacity vanishes. 



4.3 Proof of Theorem H 



Proof. Under MPI decoding, the decoding complexity of the sequence of codes {C m } is equal to 
the number of edges in the Tanner graph of the original codes {C' m } normalized per information 
bit (since for the BEC, one can modify the MPI decoder so that every edge in the Tanner graph 
is only used once). This normalized number of edges is directly linked to the average degree of 
the parity-check nodes in the Tanner graphs of the sequence of codes {C' m } (up to a scaling factor 
which depends on the rate of the code). We will first derive an information-theoretic bound on the 
average degree of the parity-check nodes for the sequence {C' m }, say aR(C' m ), which will be valid 
for every decoding algorithm. From this bound, we will directly obtain a bound on the decoding 
complexity of punctured codes on graphs, when we assume that an MPI decoding algorithm is 
used. 

Let u' m = (ui,U2, . . . ,u nm ) be a codeword of a binary linear block code C' m , and assume that 
a subset of the information bits of the code C' m are punctured (see footnote no. 5 in p. 7). Let us 
replace the punctured bits of u' m by question marks, and let us call the new vector u m . The bits of 
u m (those which were not replaced by question marks) are the coordinates of the codewords of the 
punctured code C m . Let us assume that u m is transmitted over a BEC whose erasure probability 
is equal to p. The question marks in the received vector v m = (vi,V2, ■ ■ ■ ,Vn m ) remain in all the 
places where they existed in u m (due to puncturing of a subset of the information bits of u^J, and 
in addition, the other bits of u m which are transmitted over the BEC are received as question marks 
with probability p or remain in their original values with probability 1 — p (due to the erasures 
of the BEC). Since by our assumption, the sequence of punctured codes {C m } achieves a fraction 
1 — e of the channel capacity with vanishing bit erasure probability, then there exists a decoding 
algorithm (e.g., ML decoding) so that the average bit erasure probability of the code C m goes to 
zero as we let m tend to infinity, and lim m — >c>0 

Rm > (1 — e)(l —?)■ Here, R m and R' m designate the 
rates (in bits per channel use) of the punctured code C m and the original code C' m , respectively. The 
rate of the punctured code (C m ) is greater than the rate of the original code (C' m ), i.e., R' m < R m . 
Let (to) designate the bit erasure probability of the digit Ui at the end of the decoding process 
of the punctured code C m . Without loss of generality, one can assume that the n m R' m first bits of 
the vector refer to the information bits of the code C' m , and the other n m (1-R' m ) last bits of 
u' m are the parity bits of C m and C' m . Let 

Tim R' m 

be the average bit erasure probability of the code C m (whose codewords are transmitted with equal 
probability), based on the observation of the random vector v m at the output of the BEC. By 
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knowing the linear block code C' m , then we get that 



(a) Hauj^g^Q 

§ s:ri^gKiv m ) 



(d) Egf-H^M) 



< ^^(P b (m)) 

where equality (a) is valid since the n m R' m information bits of the linear block code C' m determine 
the n m (l — R' m ) parity bits of its codewords, equality (b) is based on the chain rule for the entropy, 
inequality (c) follows since conditioning reduces the entropy, inequality (d) follows from Fano's 
inequality and since the code C m is binary, and inequality (e) is based on Jensen's inequality and 
the concavity of the binary entropy function h(x) = — xlog 2 (x) — (1 — x) log 2 (l — x) for x G (0, 1). 
Based on our assumption that there exists a decoding algorithm so that the bit erasure probability 
of the sequence of codes {C m } vanishes (as m — ► oo), then it follows that 

hm ff( < |Vm) = 0. (44) 

m-»oo rim 



For the sake of notational simplicity, we will replace u^, v m and n m by U', V, and n, respec- 
tively. In the following derivation, let K and E designate the random vectors which indicate the 
positions of the known and punctured/erased digits in the received vector (V), respectively (note 
that knowing one of these two random vectors implies the knowledge of the other vector). The 
random vector Vk denotes the sub- vector of V with the known digits of the received vector (i.e., 
those digits which are not punctured by the encoder and not erased by the BEC). Note that there 
is a one-to-one correspondence between the received vector V and the pair of vectors (Vk,E). 
We designate by U' E and U' K the sub-vectors of the original codeword U' of the code C' m , such 
that they correspond to digits of U' in the punctured/erased and known positions of the received 
vector, respectively (so that U' K = Vk). Finally, let H'^ denote the matrix of those columns of H' 
(a parity-check matrix representing the block code C' m ) whose variables are indexed by E, and |e| 
denotes the number of elements of a vector e. Then, we get 

if(U'|V) = tf(U'|V K ,E) 

= #(U' E ,U K |V K ,E) 

= H(U' E |V K ,E) 

= Ev k) eP(vk,e)ff(U' E |V K =v k ,E = e) 

= E Vkl eP(vk,e) (|e|-rank(^)) 

= EeP(e) (|e| - rank(^)) 

= EeP( e ) l e l - EeP( e ) rank(i^) . 
and by normalizing both sides of the equality w.r.t. the block length (n), then 

ffi^ = \ E^ e ) M - \ E^ e ) rank M • (45) 



17 



Note that the rank of a parity-check matrix H' e of the block code C' m is upper bounded by the 
number of non-zero rows of H' e which is equal to the number of parity-check nodes which involve 
punctured or erased bits (the sum Y2 e P( e ) ' ran k(ffe) * s therefore upper bounded by the average 
number of parity-check sets which involve punctured or erased bits). 

Now, we will bound the two sums in the RHS of (|45jl: let I m and P m be the number of 
information bits and parity bits in the original code C' m . Then n m = I m + P m is the block length of 
the code C' m , and the block length of the code C m (i.e., the block length after puncturing a fraction 
f pct of the information bits in C' m ) is equal to I m (l — fpct) + Pm- The rate of the punctured code 
is therefore equal to R m = -j—^ — i m , , p . Its asymptotic value (as m — > oo) is by assumption at 
least (1 — e)(l —p), i.e., 

lim R m = lim — — m > (l-e)(l-p). 

m-»oo m->oo l m { \ - F pct ) + P m 

We obtain from the last inequality that 

lhn In L> d-e)(l- P ) 



rn^oo P m P eS + g(l - P eff ) 

where f e ff was introduced in l)2U[) . so the asymptotic rate of the sequence of codes {C' m } satisfies 
lim R' = lim Im > w C 1 -^ 1 "?) (46) 

m ^ ™ m -+oo I m + P m ~ (l- £ )(l-p)+P eff + £ (l-P eff ) V > 

The number of elements of a vector e indicates the number of bits in the codewords of C' m which 
are punctured by the encoder or erased by the BEC. Its average value is therefore equal to 



Im -fpct + yl-miX ~ -fpct) + Pm )P 
ImP 

P 



-fm -fpct 



n mR m ( 



fpct + p 

so since R m < 1 — p, then we obtain from ()46|l and the last equality that 

lim —y P (e) |e| > i 1 ~ ^ ~ p) ( Ppct + ^ . (47 ) 

m^oo n m ~* _ (1 — e )(l — P) + feff + e (l — feff) \ 1 — p J 1 j 

If a parity-check node of the Tanner graph of the code C' m is connected to information nodes by 
k edges, then based on the assumption that the information bits of the codes in {C' m } are randomly 
punctured at rate f pc t , then the probability that a parity-check node involves at least one punctured 
or erased information bit is equal to 1 — (1 — f e ff) fc . This expression is valid with probability 1 
(w.r.t. the randomly chosen puncturing pattern) when the block length tends to infinity (or in the 
limit where m — > oo). We note that f e g is introduced in (|20|). and it stands for the effective erasure 
probability of information bits in the code C' m when we take into account the effects of the random 
puncturing of the information bits at the encoder, and the random erasures which are introduced 
by the BEC. The average number of the parity-check nodes which therefore involve at least one 
punctured or erased information bit is equal to n m (l — R' m ) d^ jin (l — (1 — f e ff) fc ) where (4 >m 
designates the fraction of parity-check nodes in the Tanner graph of C' m which are connected to 
information nodes by k edges. Therefore 



— 5>(e) rank(P4) < (1 - R' m ) ( 1 - £ d h , m (l - f c 



n m c 
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From Jensen's inequality, we obtain that 

J>fc, m (l - Pes)" > (1 " Peff) MC) 

where b^,(C' m ) = kd^ m is the average number of edges which connect a parity-check node with 
information nodes in the Tanner graph of the code C' m . By definition, it follows immediately that 
aji(C' m ) > b^(C' m ) + / m j n , and therefore we get 

-Vp(e) rank(^) < (1 - R'J (l - (1 - P eS )**( c ^-^A . (48) 



From Eqs. (JUJ-ljlHJ) , we obtain that 



^ U ' n * Vr ") = lim J_y^p(e)|e|- lim — V p(e) rank^' 

rj m — >on rj ^ — ^ m — ton rj..__ ^- — ^ 



lim 

m— »oo n m m— >oo n m z — ' " m— >oo n m 



e 



^ (I~g)(l-P) ( J~y , P 

£1 — ~ s . ^ ; — 77 — ~ I -fpct + 



(1 - e)(l - p) + P cS + e(l - P cS ) V P 1-P 

1 (l-£)(l-p) \ /^.p ^-U 



where or = liminfm^oo aR,(C^J. The limit of the normalized conditional entropy in (|44j) is equal 
to zero, so its lower bound in the last inequality cannot be positive. This yields the inequality 



(1 - p)(l - e)P pct + (1 - e)p - (P cfT + e(l - P eff )J (l - (1 - P c fT) aR ~ Un ) < 0. (49) 
From (|2l)|). then (1 — p) P pct + p — P e ff = 0, so simplification of the LHS in gives 
P e ff (1 - P eff ) aR ~ Un < e(l - p)P pct +ep + s(l - P eff ) fl — (1 — P cfr ) aR -' m 



< e{l - p)P pct + ep + e(l - P cS ) 
= e. 

This yields the following information-theoretic bound the asymptotic degree of the parity-check 
nodes (a R (C' m )) 

ln (¥) 

liminf a^(C' m ) > — -, c- + l min , (50) 

m->oc In 



t-Peff 

which is valid with probability 1 w.r.t. the puncturing patterns. The proof until now is valid under 
any decoding algorithm (even the optimal MAP decoding algorithm), and in the continuation, we 
link our result to MPI decoding. 

From the information-theoretic bound in (|5()j) . it follows that with probability 1 w.r.t. the 
puncturing patterns, the asymptotic decoding complexity of the sequence of punctured codes {C m } 
satisfies under MPI decoding 



liminf XD(Cm) = ( 1 „ ) liminfa R (C^ 

m^oo \ it J m— >oo 



where R is the asymptotic rate of the sequence {C m }. The scaling by is due to the fact that 
the complexity is (by definition) normalized per information bit, and the average degree of the 
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check nodes is normalized per parity-check node). As said before, the last equality is true since the 
MPI decoder can be modified for a BEC so that every edge in the Tanner graph is only used once; 
therefore, the number of operations which are performed for MPI decoding of the punctured code 
C m is equal to the number of edges in the Tanner graph of the original code C' m . Since R < 1 — p, 
then we obtain from (|5(J|) that under MPI decoding, the asymptotic decoding complexity satisfies 
(|19|) with probability 1 w.r.t. the puncturing patterns. 

If P pct = l-O(e), then it follows from that also P cS = I- 0(e). Therefore, the RHS of lfT3|) 
remains bounded when the gap (in rate) to capacity vanishes (i.e., in the limit where e — * 0). We 
conclude that with probability 1 w.r.t. the puncturing patterns of the information bits, a necessary 
condition that the sequence of punctured codes achieves the capacity of the BEC with bounded 
complexity under MPI decoding is that the puncturing rate of the information bits satisfies the 
condition P pct = 1 — 0(e). Otherwise, the complexity grows like O (in (^)). □ 

Discussion: Note that a-fortiori the same statement in Theorem |21 holds if we require that 
the block erasure probability tends asymptotically to zero. We note that this statement is valid 
for every sequence of codes, as opposed to a lower bound on the decoding complexity of IRA 
ensembles on the BEC which we originally derived based on the density evolution (DE) equation. 
Considering ensembles of non-systematic IRA codes with random puncturing of the information 
bits, the lower bound on the complexity that we derived from the DE equation (based on a natural 
generalization of the derivation of the bound in [151 Theorem 1] to the case of randomly punctured 
IRA code ensembles) was a slightly looser bound than the bound in Theorem 01 so we omit its 
derivation. The lower bound on decoding complexity of capacity-achieving codes on the BEC is 
especially interesting due to two constructions of capacity-achieving IRA ensembles on the BEC 
with bounded complexity that were introduced in the first two theorems of our paper. We note 
that for ensembles of IRA codes where the inner code is a differential encoder, together with the 
choice of puncturing systematic bits of the IRA codes and the natural selection of the information 
bits as the systematic bits, then l m \ n = 2 (since every parity-check node is connected to exactly 
two parity bits). For punctured IRA codes, the lower bound in (|19l) is also a lower bound on the 
encoding complexity (since the encoding complexity of IRA codes is equal to the number of edges 
in the Tanner graph per information bit, so under MPI decoding, the encoding and the decoding 
complexity of IRA codes on the BEC are the same). 

The lower bound on the asymptotic degree of the parity-check nodes in 1)50(1 is valid under 
ML decoding (and hence, it is also valid under any sub-optimal decoding algorithm, such as MPI 
decoding). Finally, the link between the degree of the parity-check nodes in the Tanner graph and 
the decoding complexity is valid under MPI decoding. 

4.4 Proof of Theorem 0] 

Proof. The proof relies on the proofs of [21 Theorem 1] and j!41 Theorem 1], and it suggests a 
generalization to the case where a fraction of the information bits are punctured before the code is 
transmitted over an MBIOS channel. 

Under MPI decoding, the decoding complexity per iteration of the sequence of codes {C m } 
is equal to the number of edges in the Tanner graph of the original codes {C' m } normalized per 
information bit. Similarly to the proof for the BEC, we will first derive an information-theoretic 
bound on the average degree of the parity-check nodes for the sequence {C' m }, say a^(C' m ), which 
will be valid for every decoding algorithm. From this bound, we will directly obtain a bound on 
the decoding complexity per iteration of punctured codes on graphs, when we assume that an MPI 
decoding algorithm is performed. 
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It suffices to prove the first bound (which refers to the limit of the average degree of the parity- 
check nodes for the sequence {C' m }) w.r.t. MAP decoding. This is because the MAP algorithm 
minimizes the bit error probability and therefore achieves at least the same fraction of capacity as 
any suboptimal decoding algorithm. According to our assumption about random puncturing of the 
information bits at rate P pc t, then it follows that the equivalent MBIOS channel for the information 
bits is given by 

q(y\x = 1) = P pct 5 (y) + (1 - P pct ) p{v\x = 1) (51) 

which is physically degraded w.r.t. the original communication channel whose conditional pdf (given 
that x = 1 is the input to the channel) is p(y\x = 1). On the other hand, since we assume that only 
information bits are punctured, then the original MBIOS channel over which the communication 
takes place is also the equivalent channel for the parity bits. 

By assumption, the sequence of punctured codes {C m } achieves a fraction 1 — e of the channel 
capacity. Let I m and P m designate the number of information bits and parity bits in the code C' m 
(before puncturing), then the rate of the punctured code C m is given by 

T 

R,, 



(1 Ppct) I'm ~i~ Pm 

According to the assumption in the theorem, we have lrnim^oo R m > (1 — e)C, which implies that 

lim *ZL > {l ~ e)C (52) 

m-oo P m ~ i-(i- e )(i- P pct )C ■ v ; 

The asymptotic rate of the original sequence of codes {C' m } (before puncturing) therefore satisfies 



m— »oc 



Similarly to the information-theoretic proof for the BEC, it follows exactly via the same chain 
of inequalities that 

lim ff(uJVm) = (54) 

m-^co n m 

where u' m and v m designate a codeword of the original code C' m , and the received vector at the 
output of the channel (after puncturing information bits from u^, and transmitting the punctured 
codeword over the communication channel), respectively. The parameter Tim designates the block 
length of the original code C' m . 

Let g(y\x = 1) be an arbitrary conditional pdf at the output of an MBIOS channel, given that 
x = 1 is the input to this channel, and let us define the operator 

i r+oc 

^(g) = -z / min(sr(y|x = l),g(y\x = 0)) dy. 

Then, it follows directly that < oj(g) < ^, and u){p) = w where w is introduced in (|2*2*)l . For the 
MBIOS channel in (JHU) 

u(q) = -7T <5o(y)dyH / mm(p(y\x = l),p{y\x = 0)) dy 



2 

so, we obtain the equality 



Ppct + (1 - P pct ) w 



l-2u{q) = (l-2w)(l-P pct ). (55) 
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Analogously to the proof of [2, Theorem 1], let us define a binary random vector Z = (z\, . . . , z nm ) 
so that for I = 1, 2, . . . , n mi if u\ and v\ designate the Z-th components of u' m and v m , respectively, 
then 

Pr(^ = 1| q(vi\ui = 1) > ?(-«j|itj = 1)) = 1 

Pr(z t = 0| = 1) < g(-«i|«i = !)) = ! 

Pr(zj = 1| q(yi\ui = 1) = g(-«i|itj = 1)) = -. 

In particular, in case that vi corresponds to an erasure, then z\ is equal to zero or one with 
probability \. Hence, the channel U' — > Z is equivalent to a BSC with crossover probability which 
is equal to u)(q). 

Based on [2| Eqs. (4), (5), (11) and (12)], we obtain that 

H(u' m \v m ) > i _ I(u' m ,v m ) _ H(S' m ) 

Urn f^m f^m 

where S' m = H' m Z T is the syndrome (we designate by H' m a parity-check matrix of the code C' m ). 
From 2, Eq. (14)], we obtain an upper bound on the normalized entropy of the syndrome for the 
case of random puncturing 

^.(l-O^ 1 -' 1 -^"' ) (5T) 

where as compared to [21 Eq. (14)], we further loosen the upper bound on the entropy of the 
syndrome by assuming that all the bits of C' m face (because of puncturing) the channel in (|51|) . In 
fact, this is true only for the information bits of the codewords of C' m (since parity bits are not 
punctured), but since the channel with the conditional pdf q(-\x = 1) is physically degraded w.r.t. 
to the original MBIOS channel, and the inequality uj(q) > w follows directly from (|55|l. then the 
upper bound in (|5T|) holds. 

Let L m designate the number of digits of the codewords in {C m } (i.e., after puncturing infor- 
mation bits at a puncturing rate P pc t), then L m = I m (l — P pc t) + P-mi an d 

/(u/m;Vm) < C (58) 

and from (|52|) . we get 

L m , ^-(1-Ppct) + 1 1 , , 

lim — = lim Fm . — < -. 59) 

,_ „ rr,-,^, lm + I - 1 + (1 _ S)P t C 



m—*oo n m m— >oo 



Pn 



Since from (|54[l. the LHS of (|56() vanishes as we let m tend to infinity, then the combination of 
Eqs. (|52|) - (|59j) gives the following inequality in the limit where m — > oo 

i c - (i ) , A-d-2^)rn <0 

l + (l- £ )P pct C V l + (l-e)P pct Cj V 2 J- 

where or = limm^oo a^(C' m ). By invoking the following inequality for the binary entropy function 
(see 01 Lemma 3.1 (p. 1618)]) 

2 / 1\ 2 1 



h(x) < 1 - - — [x - - , 0<x< . 
v; ~ ln2V 2/' - - 2 
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we obtain from the last two inequalities that 



i 1 (i (Izjg \ A. d-M^- U,, 

l + (l-£)Ppc t C V 1 + (1 - E)-PpotC/ \ 21n2 / 

We note that the last inequality extends the inequality in 14, Eq. (32)] to the case where we 
allow random puncturing of the information bits at an arbitrary puncturing rate (if there is no 
puncturing, then P pct = and u(q) = w from (|55|). so Eq. (32)] follows directly as a particular 
case). From the last inequality and (|55|) . we obtain that with probability 1 w.r.t. the puncturing 
patterns, the following information-theoretic bound on the asymptotic degree of the parity-check 
nodes is satisfied 

l n ( I i-(i--Ppct)C 



,/ \ 2Cln2 , 

hmmf a-R,(C m ) > -, r- . (60) 

m ^°° 9 ln I - I 

The proof until now is valid under an arbitrary decoding algorithm (i.e., under MAP decoding, 
and hence, under any other decoding algorithm). In order to proceed, we refer to MPI decoding, 
where the asymptotic decoding complexity per iteration of the punctured codes C m is equal to 
times or where R = linim^oo R m is the asymptotic rate of the sequence of punctured codes 
{C m }. By assumption R > (1 — e)C, so we obtain from (|fiT)|) the lower bound on the decoding 
complexity per iteration of punctured codes which is given in (|21|) . This lower bound drives us to 
the interesting conclusion that if the puncturing rate of the information bits is strictly less than 1, 
then the decoding complexity per iteration must grow at least like ln Q). On the other hand, if 
-fpct = 1 — 0(e), then the numerator and denominator of the RHS in 1)211) are both in the order of 
ln (i), and therefore the lower bound on the decoding complexity per iteration stays bounded as 
the gap (in rate) to capacity vanishes. □ 



Discussion: Like Theorem 03 the lower bound on the decoding complexity in Theorem 0] also 
clearly holds if we require vanishing block error probability. For a BEC with erasure probability 
p, the parameter w in p2|) is equal to w = |, and the capacity is equal to C = 1 — p. From (|2fl|) . 
this implies that for the BEC, (1 — 2u>)(l — P pc t) = 1 — -Peff- It therefore follows that the lower 
bound in Theorem is at least twice larger than the lower bound for the BEC which we get from 
the general bound in Theorem|lJ The derivation of Theorems 01 and |1] are also different, so because 
of these two reasons, we derive the stronger version of the bound for the BEC in addition to the 
general bound for MBIOS channels. The comparison between the two bounds for punctured codes 
on graphs is consistent with the parallel comparison in |14| Theorem 1] for non-punctured codes. 
The bounds in Theorems El and prefer both to random puncturing, so these two theorems are valid 
with probability 1 w.r.t. the puncturing patterns, if we let the block length of the codes go to 
infinity. Since these bounds become trivial for some deterministic puncturing patterns, it remains 
an interesting open problem to derive information-theoretic bounds that can be applied to every 
puncturing pattern. 
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5 Analytical Properties and Efficient Computation of the D.D. 



In this section, we tackle the problem of computing the d.d. coefficients for both the bit-regular 
and check-regular ensembles. While doing this, we also lay some of the groundwork required to 
prove that these coefficients are non-negative. Asymptotic expressions for the coefficients can also 
be computed rather easily in the process. We start with bit-regular ensemble because the analysis 
is somewhat simpler. 



5.1 The Bit-Regular Ensemble 

5.1.1 A Recursion for the D.D. Coefficients {p n } 

We present here an efficient way for the calculation of the d.d. coefficients {p n }, referring to the 
ensemble of bit-regular IRA codes in Theorem^ To this end, we derive a very simple recursion for 
coefficients of the power series expansions of R(x) and p(x). We start with Eq. (|28j) and rearrange 
things to get 

R(x) = — !— Q(x) - -2— R(x)Q(x). 
1 — p 1 — p 

Next, we substitute the power series expansions for R(x) and Q(x) to get 

oo ^ oo oo oo 

y~i Rnxn = i_ v Q nxn ~ i_ ^2 Rix% Qj xj - 

n=2 1 n=2 y i=2 j=2 

Matching the coefficients of x n on both sides gives 

I p "~ 2 

Rn = Z Qn — Z / RiQn-i- (61) 

1 — P 1 — P 



* i=2 



Using Eq. (|27|l. we can write 

Q(x) 



j;(i-(i-ov(^))dt 
/o i (1 _ (1 _ t) i/( g -i) )d t 

j J Er=i(-i) fc+1 (Y)^ 
i 



k=l 



k J k+l 



which implies that 



Qn= (_jTqf q _! j n > 2 _ (62) 
n \n — lj 



From (jnU) and it follows that 

s)_^riut2r±( * Y „> 2 . 

- 1/ 1 — p n — i \n — i — 1/ 

Since = R'(x)/R'(l), this gives 



n(l-p) V" « , 2 



^ fl'(l) 9 (l-p) 
where the last equality follows from Eqs. © and (|26|). 
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5.1.2 Asymptotic Behavior of p n 

In this section, we consider the asymptotic behavior of the coefficients in the power series expansion 
of Eq. (|1U|) . The resulting expression provides important information about the decay rate of the 
coefficients. It also shows that p n always becomes positive for large enough n and therefore lends 
support to Conjecture ^ We approach the problem by first writing p(x) as a power series in 
(1 — x) 1 /^ -1 ) and then analyzing the asymptotic behavior of each term. This approach is motivated 
and justified by the results of |3j. 

We start by rewriting Eq. IjlOjl in terms of u = (1 — x) 1 ^ 9-1 ^, and then expanding the result 
into a power series in u to get 

l-u 

p(x) 



(l-p(qui- 1 -{q- l)ui)^j 

oo 

= (l-u) ^(Hiylgn^-fe-l)^ 

= l-u + 2pqu q ~ 1 - 2p{2q - l)u q + 2p{q - l)u q+1 + 3p 2 q 2 u 2q - 2 + O {u 2 ^ 1 ) . 
Now, we can convert this into an asymptotic estimate of p n by using the fact (from [3|) that 

n- 1 — /, , a(a + l) , „ , 



where [x k ] A(x) is the coefficient of x k in the power series expansion of A(x). We note that the terms 
u q ~ l and u 2q ~ 2 are actually polynomials in x and do not contribute to the asymptotic behavior. 
Combining the remaining terms with the equality pn+i = [x n ]p(x) shows that 

(. , q 2 W (2,-1) , 4pfa+l)r(gj) _ , 

"« = ( ,_ 1) r ( ^ ) ( v 1 + %^- 1? rT>r + r(0)n - +0 (" )l- < 63 > 

We note that Gamma functions with negative arguments were modified to have positive arguments 
using the identity T(x) = T(x + l)/x. 



5.2 The Check-Regular Ensemble 

5.2.1 Three Representations for the D.D. Coefficients {A n } 

We present here three different useful expressions for the computation of the d.d. coefficients {A n }, 
referring to the ensemble of check-regular IRA codes in Theorem |^1 The first expression is based 
on the Lagrange inversion formula (see, e.g., Section 2.2]). The second expression follows by 
applying the Cauchy residue theorem in order to obtain the power series expansion of A(-) in (|15|) . 
and the third expression provides a simple recursion which follows from the previous expression. 
These three expressions are proved in Appendix B, and provide efficient numerical methods to 
calculate the d.d. of the information bits (from the edge perspective). 

First expression: 

Xn = — !-r [x n ~ 2 ] 0™- 1 (x) , n = 2,3,... (64) 
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where 

<f>(x) 4 . (65) 

This representation of the sequence {A n } follows directly from the Lagrange inversion formula by 
writing y4>{x) = x where y = A -1 (a;). 

Second expression: 

4(1 -pf \ f°° g(r) , 

X n = Im < / -^n dr } , n = 2, 3, . . . (66) 

9tt P W (i +c ( p ) re -^ 



where 



2 



c(p)=(^^^) 3 , 0<p<l (67) 



and the complex function g(-) is 



sin ( | arcsin(r4 + iw 



g{r) = lim j , r > 0. (68) 



Third expression: 



_ 2(n-l) 

Xn+lip) = (1 + 2p)^-i • £ ' " = 1» 2, . . . (69) 



i=0 



(- , v -.2(71-1) 

where for n > 2, we calculate the coefficients |°i j. with the recursive equation 

(2m - 3i - 1) (a<"» - 2a<™>) + (20m - 3i - l)<fi 
„<»«> = S, 25n+l) ' m= 1,2,... ,n- 1 (70) 

and the initial condition = ^. We define aj fc ^ as zero for i < or i > 2(fc — 1) (where 
fc = 1,2, . . .). Based on Eqs. ^ and l|7D|>. it follows that 



Hp) 
Hp) 
Hp) 
Hp) 



1-p 

2(1 + 2p) 

(1 -p)(l + 16p + 10p 2 ) 
8(1 + 2p) 3 

(1 - p)(l + 12p + 168p 2 + 164p 3 + 60p 4 ) 
16(1 + 2pf 

(1 - p)(5 + 80p + 470p 2 + 7840p 3 + 9640p 4 + 5920p 5 + 1560p 6 ) 

128(1 + 2pf 



and so on. Eqs. (|69|) and (|7()|) provide a simple way to calculate the sequence {A n } without the need 
to calculate numerically complicated improper integrals or to obtain the power series expansion of 
a complicated function; this algorithm involves only the four elementary operations, and hence can 
be implemented very easily. 
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An alternative way to express Eqs. and (|7U|) is 



(l-p)-P n (p) 
K+l(P) = -TTT TT^TT" n = l,2, ... 



[l + 2p) 



(71) 



where {-Pn(^)}„>i is a sequence of polynomials of degree 2(n — 1) which can be calculated with the 
recursive equation 



[(14 - An)x 2 + (20n - 4)x + 2n - l] P n (x) - 3x(l + s - 2x 



P n +i(x) 



and the initial polynomial Pi(x) = h. 



2^ dP n (x) 
dx 



2(n + l) 



n = 1,2, 



(72) 



5.2.2 Asymptotic Behavior of \ n 

We show in Appendix B that for a fixed value of p, the asymptotic behavior of the d.d. {A n } is 
given by 



A 



_ 3 

n 2 



n+l 



2V5F(1- P ) 



1 + 



3\/2 -(n-i) 



2 p 



sm n 



1 



3 25 / 1 



8n 128n 2 



where 



1 + e s c(p) 



9 P = arg 



1 + e 3 c(p) 



(73) 



(74) 



and c(p) is given in ()67l) . Unless p is close to one, it can be verified that the asymptotic expression for 
the coefficients {A n } also provides a tight approximation for these coefficients already for moderate 
values of n. For example, if p = 0.5, then the asymptotic expression in (|73|) is tight for n > 20 
(A20 is equal to 0.0100 while the asymptotic expression in (|73|) is equal to 0.0107). If p = 0.8, the 
asymptotic expression for the coefficients {A n } in (|73j) is tight only for n > 120 (the approximation 
for A120 is equal to 0.0020, and its exact value is 0.0021). In general, by increasing the value of p 
(which is less than unity), the asymptotic expression in l|73|) becomes a good approximation for the 
coefficients A n starting from a higher value of n (see Fig. |3J • 




Figure 3: The exact and approximated values of A n +i (n = 1, . . . , 1000) for the check-regular IRA 
ensemble in Theorem |2I The approximation is based on the asymptotic expression in (|73|) . The 
plots refer to a BEC whose erasure probability is p = 0.2 (left plot) and p = 0.8 (right plot). 
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5.2.3 Some Properties of the Power Series Expansion of A(x) in Eq. |)T""") 

The values of P n ( - ) a t the endpoints of the interval [0, 1] can be calculated from the recursive 
equation ((72*)) (the coefficient of the derivative vanishes at these endpoints). Calculation shows that 

P, l( o ) = ^_i-( 2 "Y PnW = *—(^\ „ =1>2 ,... (75) 

Since these values are positive, it follows from (|7T|) that {A n (p)} n >2 are positive for < p < 1 if 
and only if the polynomials P n {-) do not have zeros inside the interval [0, 1] for all n > 1. We note 
that if for every n > 1, all the coefficients of the polynomial P n (-) were positive, then based on 
(fTTj) . this could suggest a promising direction to prove the positivity of {X n (p)}n>2 over the whole 
interval p S [0,1). Unfortunately, this is only true for n < 6. The positivity of {A„(p)} n >2 over 
the interval [0,0.95] is proved in Appendix C, based on their relation to the polynomials P n (')- As 
we already noted, our numerical results strongly support the conjecture that this is true also for 
p e (0.95,1). 

As we will see, the behavior of the functions A n (p), in the limit where p — > 1 and n — » oo, 
depends on the order that the limits are taken. When the value of n is fixed, it follows from (|71|) 
and 1)75)1 that in the limit where p — > 1 

, , , P n -i(l) s 1 1 /2n-2\ , 

( 1 -P) = 3i^ T ( vn _ 1 J - = 2,3,4,... (76) 

Therefore, for a fixed value of n, A n (p) is linearly proportional to 1 — p when p — » 1. On the other 
hand, if the value of p is fixed (0 < p < 1) and we let n tend to infinity, then we obtain from (|73|) 
that A n (p) is inversely proportional to 1 — p. Observe from Fig. U3 that the sequence of functions 
{X n (p)}n>2 is monotonically decreasing for all values of p, and that the tail of this sequence becomes 
more significant as the value of p grows. This phenomenon can be explained by Eq. 1)73)1 since the 
asymptotic behavior of the sequence {X n (p)} is linearly proportional to this makes the tail of 
this sequence more significant as the value of p is closer to 1. It seems from the plots in Fig. [3 that 
the asymptotic expression ()73*|) forms an upper bound on the sequence {X n (p)}. From these plots, 
it follows that by increasing the value of p, then the approximate and exact values of X n (p) start 
to match well for higher values of n. We note that the partial sum 

En=2 X n(p) is equal to 0.978, 

0.970, 0.955 and 0.910 for p = 0.2, 0.4, 0.6 and 0.8, respectively; therefore, if the value of the erasure 
probability (p) of the BEC is increased, then the tail of the sequence {A n (p)} indeed becomes more 
significant (since the sum of all the A n (p)'s is 1). More specifically, we use the fact that 

y>- a = (1 + (1)), a >l 

and rely on Eq. lff3*|) in order to show that for a large enough value of N, the sum ^^=2 ^n(p) is 
approximately equal to 1 — ^ l N j^', this matches very well with the numerical values computed 
for N = 1000 with p = 0.2 and 0.8. 
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6 Practical Considerations and Simulation Results 



In this section, we present simulation results for both the bit-regular (Theorem ^) and check- 
regular (Theorem |2| ensembles. While these results are provided mainly to validate the claims of 
the theorems, we do compare them with one other previously known ensemble. This is meant to 
give the reader some sense of their relative performance. Note that for fixed complexity, the new 
codes eventually (for n large enough) outperform any code proposed to date. On the other hand, 
the convergence speed to the ultimate performance limit is expected to be quite slow, so that for 
moderate lengths, the new codes are not necessarily expected to be record breaking. 



6.1 Construction and Performance of Bit-Regular IRA Codes 

The bit-regular plot in Fig. compares systematic IRA codes [3] with X(x) = x 2 and p(x) = x 36 
(i.e., rate 0.925) with bit-regular non-systematic codes formed by our construction in Theorem^ 
with q = 3. This comparison with non-systematic IRA codes was chosen for two reasons. First, both 
codes have good performance in the error floor region because neither have degree 2 information 
bit. Second, LDPC codes of such high rate have a large fraction of degree 2 bits and the resulting 
comparison seemed rather unfair. We remind the reader that the bit-regular ensembles of IRA 
codes in Theorem ^ are limited to high rates (for q = 3, the rate should be at least j| ~ 0.9231). 
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N=64000 M=50 (Irreg) 
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N=500000 (Reg) 
N=64000 (Reg) 
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N=8000 M = 75 (Irreg) 
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N=8000 (Reg) 
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Figure 4: BER and WER for random rate 0.925 codes from the bit-regular IRA ensemble in 
Theorem^with q = 3 and the regular systematic IRA ensemble with d.d. X(x) = x 2 and p(x) = x 36 . 
The curves are shown for N = 8000, 64000, and 500000. 

The fixed point at x = 1 of the DE equation for the ensemble in Theorem ^ prevents the decoder 
from getting started without the help of some kind of " doping" . We overcome this problem by using 
a small number of systematic bits (100-200) in the construction. Of course, these bits are included 
in the final rate of 0.925. Codes of block length N = 8000, 64000 and 500000 were chosen from 
these ensembles and simulated on the BEC. The parity-check d.d. of each bit-regular code was 
also truncated to maximum degree M. All codes were constructed randomly from their d.d.s while 
avoiding 4 cycles in the subgraph induced by excluding the code bits (i.e., w.r.t. the top half of 
Fig.QJ. 

It is clear that the bit-regular ensemble performs quite well when the block length is large. As 
the block length is reduced, the fraction of bits required for "doping" (i.e., to get decoding started) 
increases and the performance is definitely degraded. In fact, the regular systematic IRA codes 
even outperform the bit-regular construction for a block length of 8000. 
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6.2 Construction and Performance of Check-Regular IRA Codes 



The performance of the check-regular construction in Theorem was also evaluated by simulation. 
A fixed rate of 1/2 was chosen and non-systematic IRA codes were generated with varying block 
length and maximum information-bit degree. For comparison, LDPC codes from the check-regular 
capacity-achieving ensemble were constructed in the same manner. This ensemble was chosen 
for comparison because it has been shown to be essentially optimal for LDPC codes in terms of 
the tradeoff between performance and complexity [01 HE]. The IRA code ensembles were formed 
by treating all information bits degree greater than M = 25, 50 as pilot bits. The LDPC code 
ensembles were formed by choosing the check degree to be q = 8, 9 and then truncating the bit d.d. 
so that A(l) = 1. This approach leads to maximum bit degrees of M = 61, 126, respectively. Actual 
codes of length N = 8192, 65536 and 524288 were chosen from these ensembles, and simulated over 
the BEC. The results of the simulation are shown in Fig. |SJ To simplify the presentation, only the 
best performing curve (in regards to truncation length M) is shown for each block length. 



N-524288 M-50 (IRA) 
N=65536 M=50 (IRA) 
N=8192 M=25 (IRA) 
N=524288 M=126 (LDPC 
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Figure 5: BER and WER for random rate 1/2 codes from the check-regular IRA ensemble in 
Theorem El and the check-regular LDPC ensemble [H] for N = 8192,65536, and 524288. 

The code construction starts by quantizing the d.d. to integer values according to the block 
length. Next, it matches bit edges with check edges in a completely random fashion. Since this 
approach usually leads both multiple edges and 4-cycles, a post-processor is used. One iteration of 
post-processing randomly swaps all edges involved in a multiple-edge or 4-cycle events. We note 
that this algorithm only considers 4-cycles in the subgraph induced by excluding the code bits (i.e., 
the top half of the graph in Fig.^). This iteration is repeated until there are no more multiple edges 
or 4-cycles. For the IRA codes, a single "dummy" bit is used to collect all of the edges originally 
destined for bits of degree greater than M. Since this bit is known to be zero, its column is removed 
to complete the construction. After this removal, the remaining IRA code is no longer check regular 
because this "dummy" bit is allowed to have multiple edges. In fact, it is exactly the check nodes 
which are reduced to degree 1 that allow decoding to get started. Finally, both the check-regular 
IRA and LDPC codes use an extended Hamming code to protect the information bits. This helps 
to minimize the effect of small weaknesses in the graph and improves the word erasure rate (WER) 
quite a bit for a very small cost. The rate loss associated with this is not considered in the stated 
rate of 1/2. 
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6.3 Stability Conditions 



While the condition is both necessary and sufficient for successful decoding, we can still gain 
some insight by studying (JJJ) at its endpoints x = and x = 1. The condition that the fixed point 
at x = be stable is commonly known as the stability condition. Our capacity-achieving d.d. pairs 
actually satisfy (J7J), but focusing on the points x = and x = 1 gives rise to just two stability 
conditions. For decoding to finish, the fixed point at x = must be stable. While, to get decoding 
started, it helps if the fixed point at x = 1 is unstable. 

The stability condition at x = can be found by requiring that the derivative of the LHS of 
(J7J) is less than unity at x = 0. Writing this in terms of A2 (assuming that A(0) = 0) gives 

+ /?'(!) 

Since capacity-achieving codes must satisfy the stability condition with equality, we see that the 
RHS of jZ3) must go to zero for capacity-achieving ensembles without degree 2 information bits. 
It is worth noting that the bit-regular ensemble in Theorem ^ (which has no degree 2 information 
bits) has p'(l) = 00 and therefore meets the stability condition with equality. 

The stability condition at x = 1 can be found by requiring that small deviations from x = 1 
grow larger. Essentially, this means that the derivative of the LHS of ((7J) should be greater than 1 
at x = 1. Writing this in terms of P2 (assuming p(0) = 0) gives 

p2 > (l-p)'A'(l)- (78) 

Since capacity-achieving codes must also satisfy this stability condition with equality, we see that the 
RHS of (|78jl must go to zero for capacity-achieving ensembles without degree 2 parity-check nodes. 
It is worth noting that the check-regular ensemble in Theor em |2] (which has no degree 2 parity-check 
nodes) has A'(l) = 00 and therefore meets the stability condition with equality. Furthermore, one 
can see intuitively how degree 2 parity-checks help to keep the decoding chain reaction from ending. 

In reality, the fixed point at x = 1 is usually made unstable by either allowing degree 1 parity- 
checks or adding systematic bits (which has a similar effect). Therefore, the derivative condition is 
not strictly required. Still, it can play an important role. Consider what happens if you truncate 
the bit d.d. of the check-regular ensemble and then add systematic bits to get decoding started. 
The fixed point at x = 1 remains stable because the truncated bit d.d. has A'(l) < 00. In fact, 
the decoding curve in the neighborhood of x = 1 has a shape that requires a large number of 
systematic bits get decoding started reliably. This is the main reason that we introduced the "pilot 
bit" truncation in Theorem |2j 



31 



7 Conclusions 



In this work, we present two sequences of ensembles of non-systematic irregular repeat-accumulate 
(IRA) codes which asymptotically (as their block length tends to infinity) achieve capacity on the 
binary erasure channel (BEC) with bounded complexity (throughout this paper, the complexity is 
normalized per information bit). These are the first capacity-achieving ensembles with bounded 
complexity on the BEC to be reported in the literature. All previously reported capacity-achieving 
sequences have a complexity which grows at least like the log of the inverse of the gap (in rate) 
to capacity. This includes capacity-achieving ensembles of LDPC codes [7||HJE]) systematic IRA 
codes [1JIII1 EH) an d Raptor codes ^7j- The ensembles of non-systematic IRA codes which are 
considered in our paper fall in the framework of multi-edge type LDPC codes |13| . 

We show that under message-passing iterative (MPI) decoding, this new bounded complexity 
result is only possible because we allow a sufficient number of state nodes in the Tanner graph 
representing a code ensemble. The state nodes in the Tanner graph of the examined IRA ensembles 
are introduced by puncturing all the information bits. We also derive an information-theoretic lower 
bound on the decoding complexity of randomly punctured codes on graphs. The bound refers to 
MPI decoding, and it is valid for an arbitrary memoryless binary-input output-symmetric channel 
with a special refinement for the BEC. Since this bound holds with probability 1 w.r.t. a randomly 
chosen puncturing pattern, it remains an interesting open problem to derive information-theoretic 
bounds that can be applied to every puncturing pattern. Under MPI decoding and the random 
puncturing assumption, it follows from the information-theoretic bound that a necessary condition 
to achieve the capacity of the BEC with bounded complexity or to achieve the capacity of a general 
memoryless binary-input output-symmetric channel with bounded complexity per iteration is that 
the puncturing rate of the information bits goes to one. This is consistent with the fact that the 
capacity-achieving IRA code ensembles introduced in this paper are non-systematic, where all the 
information bits of these codes are punctured. 

In Section|f)J we use simulation results to compare the performance of our ensembles to the check- 
regular LDPC ensembles introduced by Shokrollahi [EJ. For the cases tested, the performance of 
our check-regular IRA codes is slightly worse than that of the check-regular LDPC codes. It is clear 
from these results that the fact that these capacity-achieving ensembles have bounded complexity 
does not imply that their performance, for small to moderate block lengths, is superior to other 
reported capacity-achieving ensembles. Note that for fixed complexity, the new codes eventually 
(for n large enough) outperform any code proposed to date. On the other hand, the convergence 
speed to the ultimate performance limit happens to be quite slow, so for small to moderate block 
lengths, the new codes are not necessarily record breaking. Further research into the construction 
of codes with bounded complexity is likely to produce codes with better performance for small to 
moderate block lengths. 

The central point in this paper is that by allowing state nodes in the Tanner graph, one may 
obtain a significantly better tradeoff between performance and complexity as the gap to capacity 
vanishes. Hence, it answers in the affirmative a fundamental question which was posed in |14j 
regarding the impact of state nodes (or in general, more complicated graphical models than bipar- 
tite graphs) on the performance versus complexity tradeoff under MPI decoding. Even the more 
complex graphical models, employed by systematic IRA codes, provides no asymptotic advantage 
over codes which are presented by bipartite graphs under MPI decoding (see |141 Theorems 1, 2] 
and ^3 Theorems 1, 2]). Non-systematic IRA codes do provide, however, this advantage over 
systematic IRA codes; this is because the complexity of systematic IRA codes becomes unbounded, 
under MPI decoding, as the gap to capacity goes to zero. 
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Appendices 



Appendix A: Proof of the Non-Negativity of the Power Series Expansion of p( ) 

in (fTI)|) 

Based on the relation between the functions R(-) and p(-), we see that p(-) has a non-negative 
power series expansion if and only if R(-) has the same property. We find it more convenient to 
prove that R(-) has a non-negative power series expansion. Starting with Eq. (J28|). we can rewrite 
R(x) as 



R(x) 



1 Q{x) 




One can verify from Eq. (j62|) that the power series coefficients of Q(x) are positive. Therefore, the 



sum 




also has a non-negative power series expansion. Based on this, it follows that the R(-) has a 
non-negative power series expansion if the function 

2 



P 



1 — p 



Q(x) 



P 



1 — p 



Q{x 



has the same property. This means that the power series expansion of R(-) has non-negative 
coefficients as long as 

2 



P 



1 



P 



Q{x) > [x k ] 



P 



1 



p 



Q(x 



k = 0,1,2, 



(A.l) 



where [x k ] A{x) is the coefficient of x k in the power series expansion of A(x). Since Q(-) has a non- 
negative power series expansion starting from x 2 , it follows that Q 2 (-) has a non-negative power 
series expansion starting from x 4 . Therefore, the condition in inequality (|A.1|) is automatically 
satisfied for k < 4. For k = 4, 5, the requirement in ()A.1|) leads (after some algebra) to the 
inequality in l|18|) . Examining this condition for q = 3 and in the limit as q — > do, we find that 
the value of p should not exceed ^ and respectively. While we believe that the power series 
expansion of R(-) is indeed positive for all p satisfying ()18|) , we were unable to prove this analytically. 
Even if this is true, it follows that R(-) has a non-negative power series expansion only for rather 
small values of p. 

For the particular case of q = 3, however, we show that the condition p < is indeed sufficient 
to ensure that (|A.1|) is satisfied for all k > 0. 
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Proof. In the case where q = 3, we find that 



Q(x) = (-2 + 2z) (l - VI - x) + x (A.2) 

and 

Q(xf = (8 - 20a; + 12x 2 ) (l - y/1 - x) - Ax + x 2 - 4x 3 . (A.3) 
Expanding l|A.2|) in a power series gives 



Q(x) = x 

and matching terms shows that for k > 2 



+ (-2 + 2*) £{(T) 



Doing the same thing for (|A.3|) shows that, for A: > 4, 



The maximal value of p such that the condition (|A.1|) is satisfied for k > is given by 

p [a:*] Q(x) 



1 — p [x k ] Q(x) 



2k-3 



g _ 20fc + 12fc(fc-l) 

2/ ''~ r ' fc = 4,5,6,... 



4(A; + 5) 

Since the RHS of this inequality is strictly increasing for k > 4, the maximal value of p which 
satisfies IjA.ljl is found by substituting k = 4. This gives the condition p < and completes the 
proof that the power series expansion of R(-) is non-negative for q = 3 and p < j^. □ 
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Appendix B: Proof of Properties of the D.D. Coefficients {A n } 



The main part of this appendix proves the three different representation for the d.d. {A n } which 
follows from the power series expansion of A(-) in (|15|), We also consider in this appendix some 
properties of the polynomials P n (-) which are related to the third representation of the d.d. {A n }, 
and which are also useful for the proof in Appendix C. Finally, we consider the asymptotic behavior 
of the d.d. {A n }. 

We start this Appendix by proving the first expression for the d.d. {A„} in (|64j) - (j65j) . 

Proof. The first representation of the d.d. {A n } in (|64j) follows directly from the Lagrange inversion 
formula by simply writing y<j)(x) = x where y = X^ 1 (x) is introduced in (j39j) . and (/)(■) = x-^( x ) * s 
introduced in (JHSJ). We note that A n is the coefficient of x n_1 in the power series expansion of the 
function X(x) in fj 1 5 D . □ 

The proof of the second expression for the d.d. {A n } in (|66|) - (|68|) relies on the Cauchy residue 
theorem. 

Proof. The function A(-) in (|15|) is analytic except for three branch cuts shown in Fig. HO The first 
branch cut starts at one and continues through the real axis towards infinity. The remaining two 
branches are straight lines which are symmetric w.r.t. the real axis. They are located along the 

lines defined by z = 1 + c(p)re :iz ~ , where c(p) = ^27^ ^ and r > 1. By the Cauchy Theorem 
we have 

X n = ^- f ^-dz, n>2 (B.l) 
Ini Jy z n 

where the contour V = T\ D T2 D T3 is the closed path which is shown in Fig. and which is 




Figure 6: The branch cuts of the function X(x) in (|15|). and the contour of integration in Eq. (|B.1|) . 

composed of three parts: the first part of integration (ri) is parallel to the branch cut at an angle 
of 60°, and it starts from the point z = 1 on the real axis and goes to infinity along this line, so it can 
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be parameterized as |ri : z = 1 + c(p) r e 3 + iw^ where < r < R and w —* + is an arbitrarily 

small positive number (the straight line T\ is parallel but slightly above the branch cut whose angle 
is 60° with the real axis). We later let R tend to infinity. The second path of integration (T2) is 

along the part of the circle of radius 1 + c(p)Re~ where we integrate counter-clockwise, starting 

from z = 1 + c(p)Re~z + iw and ending at z = 1 + c(p)Re~^~ — iw. The third part of integration 
is a straight line which is parallel to the branch cut whose angle with the real axis is —60°, but is 
slightly below this branch cut; it starts at the point z = 1 + c(p)Re~~ — iw, and ends at the point 
z = 1 on the real axis (so the points on T3 are the complex conjugates of the points on T±, and the 
directions of the integrations over T± and T3 are opposite). Overall, T = Fi UT2 L)T^ forms a closed 
path of integration which does not contain the three branch cuts of the function A(-) in ()15|) . and it 
is analytic over the domain which is bounded by the contour T. We will first show that the above 
integral over T2 vanishes as the radius of the circle tend to infinity, so in the limit where R — > 00, 
we obtain that the integral over T is equal to the sum of the two integrals over T% and l^V In order 
to see that, we will show that the modulus of X(z) — 1 in ()15|) is bounded over the circle \z\ = R 
(when R is large), so it will yield that for n > 2, the integral A ^ -1 dz vanishes as R — > 00. To 
this end, we rely on the equality 







Q arcsin(z) 






-5 



IZ 



+ V1 



IZ 



3 



which yields that |sin (^ arcsin(z))| = O 



1 

z\ 3 



By substituting z' = 



Pi 1 -^ then 
(-1+p) 3 ' iaen 





"l - / 


sin 


— arcsm 




3 \ 




o 



z 3 



o 



1 

z\ 4 



and from ()15|) . 
\X(z)-l\ = 



4(1 -P) . I 1 

, sin — arcsm 

3pVT^z \ 3 



27p(l - zf[ 2 
4(1 -p) 3 



O 



1 

z\ 4 



O(l). 



From the last equality, it follows that 



lim 

R^oo 



\(z) - 1 



dz = 0, Vn>2. 



(B.2) 



r 2 



Now we will evaluate the integral over the straight line Ti (and similarly over T3). Let 

w -> + , r > 



, 4 (1-pY _ 
z = 1 + I — — ) re 3 + iw, 



27 p 

then after a little bit of algebra, one can verify that 

1 2*i . ( 1 . / 2 
'4\ 3 e 3 sin ( 3 arcsm(r4 

so from the parameterization of Ti and l|B.3|) . we obtain that 

X(z) - 1 



\(z) - 1 



lim 



dz 



g(r) 



4\ 3 /•+°° 
- c(p) / — 
P/ Jo M+c(p)re" 



dr 



(B.3) 



(B.4) 
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where c(p) is introduced in ()67|). and the function g(-) is introduced in ()68[). 



Since the points on I^ are the complex conjugates of the points on Ti, and the integrations over 
T\ and 1^ are in opposite directions, then it follows from ()B.4|) that 



lim 



X(z) - 1 



dz 



i 

4\ 3 



c(p) 



i?-oo./ r3 z- \pj j f l + c (p) r e -T- 



+oo 



dr. 



(B.5) 



By combining Eqs. (|B. 1|) (|B.5|) and (|67() [we note that c(p) in ()67l) is real for < p < 1], we obtain 
that for n > 2 



1 



An — - — : 



A(*) 



1 



d* 



/• A(z)-1 , /■ A(z)-1 
, hm / — ^ dz + lim / dz 



4\3 c(p) 

p y 27ri 

4y c(p) 

p y 2vrz 



z" 

+oo 



g*(r) 



1 + c(p) re 3 

+oo 



-2i) Im 



dr 



9(r) 



+oo 



g(r) 



o 



c(p) 



IT \p 



1 

4\ 3 



Im 



+oo 



1 + c{p) r e 3 
9{r) 



1 + c(p) res 
dr 



dr 



9vrp | jo 



1 + c(p) r e s 



dr 



dr 



1 + c(p) res 

which coincides with the representation of X n in l|fifij ) -(|68j ) . 



□ 



The Proof of the third expression for the sequence {A n } in (|69|) — (|75|) is based on the previous 
expression which was proved above, and it enables to calculate the d.d. {A n } in an efficient way. 

Proof. From equation which we already proved, then we obtain that 

4(1 -P? 



k=n+l 



97rp 

4(1 -pf 
9np 

4(1 -p) 2 
9irp c(p) 



( ) 

Im t / g(r) ^ — ^ dr \ 

Jo k=n+i (l + c(p) re^J J 



Im < 



• Im 



+oo 

r 

+0O 



l+c(p) re'f 1 + C(P) r e" 



n+1 



dr 



5(0 



py 7T 



r e 3 (^1 + c (p) res 



— / / \ \ 

res ( 1 + c(p) r e s I 



dr 



w dr 
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where the last transition is based on Q67|) . Therefore, by multiplying both sides of the last equality 
by (|)^ and differentiating with respect to p, we obtain the equality 

1 oo 



k=n+l 



- — ■ Im 

7T 



n 



dc 



■ Im < 



o 



7T dp 

n dc 9np 



C,{ ' ' J- ( (1 + c(p) r eT 



dr 



a re™ <^P 

+oc g(r) 



dr 



1 + c(p) res 



■ A„+i(p) 



vr dp 4(1 -p) 2 

where the last transition is based on (j66ft . The function c(-) introduced in (|67ft is monotonic 
decreasing in (0, 1], and 

9c _ 2! (l-p)(l + 2p) 

#P 27 pi 

so by combing the last equality with the previous one, it follows easily that 



< p < 1 



where 



d_ J /p\ 1 

9p 



a(p) 



E Xk(p)> =n a(p) X n +i(p) 

k=n+l ) 

9p dc 25 l + 2p 1 



4(1 — p) 2 dp 



3 1 -p 



(B.6) 



(B.7) 



P" 



From ()B.6|) . we get that for n > 2 



d_ \ (P\l 
dp 



Kip) 



d J (p 

dp 



1 00 
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E A <^) 



k=n 



d \ (p 

dp 



1 00 
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E Afc ^ 

fc=n+l 



= a(p) (n - 1) X n (p) - n X n +i(p) 
so, the following recursive equation follows for n > 2 

1 d f fp\\ 



K+i(p) 
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(n 
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(a) 1 
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n 





(n - l)A n (p) 



a(p) <9p 



An(p) 



1 1-p 

2 1 + 2p 



1 



2(1 + 2p)n 



(4n - 3)p + (2n - 3)) A n (p) - 3p(l - p)A^(p) 



where transition (a) is based on ()B.7|) . and the last two transitions involve a little bit of algebra. 
In order to start the recursive equation from n = 1, we write it as 

[(An + l)p + (2n - 1)] A n+1 (p) - 3p(l - p)A' (p) 



An+2(P) 

with the initial value A2(p) 
it can be shown that 



n = l,2,... 



(Bi 



2(l + 2p)(n+l) 

1-p Based on the recursive equation (|F3.8|) and the value of A2O 



2(l+2p) ' 



Aa(p) 
A 4 (p) 



(1 -p)(l + 16p + 10p 2 ) 

8(1 + 2p) 3 
(1 - p)(l + 12p + 168p 2 + 164p 3 + 60p 4 ) 
16(1 + 2p) 5 
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and so on, which suggests the possible substitution X n+ i(p) = fef ^p^i for a certain sequence of 
polynomials {-Fn(')}n>r Combining the last substitution with (|B.8|) gives rather easily the recursive 
equation (|72|). and it therefore justifies the existence of such polynomials P n (-) in (fTTj) which can 
be calculated recursively from (|72[) with the initial polynomial -Pi(-) = \ (the initial polynomial is 
determined from the value of A2(-))- in general, it is easy to verify from (|72|) that for n > 1, P n (-) 
is a polynomial of degree 2(n — 1). Eq. (|75|) follows easily from ()72|) . as if we substitute p = or 
p = 1 in (|72|) . then the coefficient of P^(-) vanishes, so we obtain that 

2n — 1 9(2n + 1) 

Pn + M = P„(0) , P n+1 (l) = P„(l) , n = 1, 2, . . . 

where -Pi(O) = -Pi(l) = \- The closed form solutions for P n (0) and -P n (l) are given in (|75|). □ 



We will prove now a property of the polynomials -P n ( - ) which will be useful for the proof of the 
positivity of the power series expansion of A(-) (see Appendix C). 

Lemma 3. If for some n G N, the polynomial P n (-) has a zero in the interval [0,1], then the 
polynomial P n+ i(-) has also a zero in the same interval. 



Proof. Since from Eq. (|75|). -P n (0) and -P n (l) are both positive, then if P n (-) has a zero in [0,1], 
then it follows that the minimal value of P n (-) over this interval is obtained at an interior point 
x n G (0, 1). So, P n {x n ) < and P n '(x„) = 0. Since (14 - 4n)x 2 + (20n - 4)x + (2n - 1) > for 
n > 1 and x G [0,1], then from Eq. JJ2J), P n+ i(x n ) < 0. From Eq. JE}, i«+l(0) and P„+i(l) 
are both positive, but P n+ \(x n ) < 0, so the continuity of the polynomial P n +i(-) yields that this 
polynomial has at least one zero inside the interval [0, 1]. As a direct consequence of this proof and 
the equality in (|71jl. we obtain that if for a certain positive integer n, the value of A n (-) is positive 
in the interval [0, 1), then for k = 2, 3, . . . , n — 1, Afe(-) should be also positive in this interval. □ 



The above observation points towards one viable way of proving the positivity of the sequence 
{A n }: show that P n (-) is strictly positive for all x £ [0,1] for n sufficiently large. The following 
heuristic argument shows that this should indeed be the case. Unfortunately though, it seems 
nontrivial to make this argument precise, and therefore we will use in Appendix C a different route 
to actually prove the positivity of the coefficients. 

Our heuristic argument goes as follows. Consider Fig. (see next page) which shows — ^— — - 

ln( P n (x)) 

for increasing values of n. This figure suggests that — ^— — - converges uniformly over the whole 
range x G [0, 1] to a smooth limit. In order to find this limiting function, let us assume that 
approximately 

PJ X ) = C n e nf{x) where lim = o. 

n— >oo n 

Then we obtain that 

P n+1 {x) = C n+l e^ n+1 ^ = C n+ ie n/(x) e«*> 
P' n {x) = C n ne n ^f'(x). 

By substituting these asymptotic expressions in (|72jl. we obtain the equation 

(C n+l \ f(x) (14-4n)x 2 + (20n-4)x + (2n-l)-3nx(l + x-2x 2 )/ / (x) 

l^rj 6 wtt) • (B - 9) 
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Figure 7: Plots of Isi^iM) f or the polynomials P n {-) which are introduced in Eq. (|72j) . In the limit 
where n — > oo, these curves converge uniformly on [0, 1] to the function f(x) = 21n(l + 2x). 



Let us assume that linij^oo C 'X +1 



where n — > oo, (E 



1 (this assumption will be verified later). Then in the limit 
yields the following differential equation 



-2x 2 + Wx + 1 - - x{l + x - 2x 2 )f'(x). 



(B.10) 



From (|B~T0l) . it follows directly that /(0) = and /(l) = ln(9). It can be easily verified that the 
solution of the differential equation (jB.lOj) is /(x) = 21n(l + 2x) which in spite of the approximation 



suggested above coincides with the limiting function of 



ln(PnQr)) 



when n tends to infinity (see Fig.EJ). 



We note that the assumption that 
sufficiently large n 



( n 



- converges to 1 can be justified from (|75|) as follows: for 



Pn(l) 

e«/(i) 



9 n 



1 1 

9 4" 



2n 
n 



O 



C 

so the limit of ^ + is equal to 1 as n tends to infinity. In a neighborhood of x = 1, we obtain from 



the solution for the asymptotic function /(•) that 

1 1 (In 



Pn(x) 



9 4 r 



(1 + 2x) 



2n 



fB.ll) 



We note that after after some straightforward (though tedious) calculations, it can be shown from 



(|72l) that P^(l) 



12-9 n ~ 2 (n-1) (2n 



( and hence it follows from (f?5]) that -P^(l) 



4(ra-l) 



Pn(l). 



However, the approximation in (|B.11|) gives that P^(l) « SL p n (i) w hich is indeed a very good 
approximation to the equality that we get from an accurate analysis (especially for large values of 
n). We also note that if we fix the value of n (for a large enough value of n), and we let p tend to 
unity (i.e., p — > 1~), then it follows from ()72|) and (|B.11|) that 



(i-p)P n ( P ) 

(1 + 2p) 



2n-l 



I 1 (2n 
3 4™ I n 



0--P) 



which coincides with (|76|) . 
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The asymptotic behavior of the d.d. {A n }: We will discuss now the asymptotic behavior of the 
d.d. {A n } which is given in Section 15,2.21 We refer the reader to P] which relates an asymptotic 
expansion of a function around a dominant singularity with the corresponding asymptotic expansion 
for the Taylor coefficients of the function. The problem in our case is that in order to get good 
approximations of the power series expansion of A(-) in (|15|) around the dominant singularity (which 
is at 1), the function can be extended analytically beyond the radius of 1 for as far as possible 
except for a cone around 1 (see the solid lines in Fig. which are the branch cuts of A(-)). The 
further one can expand the function around the dominant singularity, the more it is determined by 
this singularity, and the Taylor series expansion around this singularity will be very accurate for 
determining the behavior of A n starting from moderate values of n. For the function A(-) in (|15j) . 
if we increase the value of p and make it closer to 1 , then unfortunately, the two other singularities 
of A(-) at z = l + c(p)e ±_ 3", where c(-) is given in (|fi7|). move towards 1 very quickly (e.g., if p = 0.8, 
then the other two singularities are located at 1.0065 ± 0.011255i). This is the reason why by 
increasing the value of p, the asymptotic expansion kicks in for larger values of n (see Fig. El- 

The proof of the asymptotic behavior of the d.d. {X n } goes as follows. If z — * 1, then the function 
A(-) in (|15|) is approximately equal to 1 — ^~ p z (since if u ~ 0, then sin(-u) ~ u and arcsin(-u) ~ u). 

/ -j\n+l / — \ 111 /2nA 

The coefficient of z n in the latter function is ^-r- I 2 ) which is equal to t*— n i -jw ( )• 

1— p \n/ 1— p in— 1 4° \ n/ 

Therefore, the contribution of the dominant singularity of A(-) to the asymptotic behavior of A n +i 
(i.e., the coefficient of z n in the Taylor series expansion of X(z)) is given by 

1 1 1 f2v> 



l-p 2n-l A n \n 

= ^L±_ [l + 3I + 251,__)| 
2y/n(l-p) I 8 n 128 n 2 1 ! ' 1 

This already gives the first term of the asymptotic behavior of A n in (|7Hjl ; based on our explanation 
above, it is a tight approximation of the asymptotic behavior of X n+ i(p) for rather small values of 




± 



3 



p. As was mentioned above, for larger values of p, the other two singularities at = 1 + c(p)e 
where c(-) is introduced in (|67|) are very close to the dominant singularity at z = 1. In order to 
determine the asymptotic behavior of \ n +i(p) for these larger values of p, we therefore need to take 
into account the asymptotic expansion of the function A(-) around these two singularities. After 
some algebra, one can verify that the behavior of the function A(-) around its singularity at z = Z\ 
is like 

J 2 -f « ^iJl-j- 
A(#i(^l+ 6 



tfZp 2V2 l-p 

and the behavior of A(-) around its second singularity at z = Z2 is like the complex conjugate of 
A(z) (since z\ and Z2 form a pair of complex conjugates). The influence of these two singularities on 
the asymptotic behavior of X n+ \{p) is therefore equal to 2 Re {[,z n ]j4(,z)}, where [z™]^^) designates 
the coefficient of z n in the power series expansion of A(z). Calculation shows that for n > 1 



where a p and 6 P are introduced in (|74jl. and for large values of n, the last expression behaves like 



3 



The sum of the two terms in ()B.12|) and ()B.13|) finally gives the asymptotic behavior of \ n +i(p) m 

CZ3>. 
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Appendix C: Proof of the Positivity of the Power Series Expansion of A( ) in (fT5|) 

The function A(-) in (|15|) is analytic except for three branch cuts shown in Fig. |HJ The first branch 
cut starts at one and continues towards infinity. The remaining two branches are symmetric around 

the real axis. They are located along the line x = 1 + c(p)re =t ^" where c(p) = ^ 4 ^ 7 ^ ) and 
r > 1. By the Cauchy Theorem we have 



where the contour can be taken e.g., along a circle of radius r, r < 1, enclosing the origin. Such a 
circle is shown as dashed line in Fig. |HJ Also shown is a modified contour of integration in which 
the circle is expanded so that in the limit the integral wraps around the three branch cuts (we note 
that the path of integration in Fig. |S]is clearly different from that one in Fig. H3). Note that the 
modulus of X(z) is bounded so that for large values of R, the integral around the circle vanishes 
as R — > oo [in Appendix B, we show that if R — > oo then \\(z)\ < O(l) on the circle \z\ = R, so 

^41 < O ( 1 ) 1 



Figure 8: The branch cuts of the function X(x) in (|15|). the original contour of integration of radius 
r, r < 1, and the modified contour with radius R tending to infinity. 

Taking advantage of the symmetries in the problem, we see after a little bit of calculus that 
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Re I 



-8(1 -p f 
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h(r) 



1 + c(p)re 3 



dr + lim — 

e^0+ TTl Ji 



1 [°° A(x + ei) 



r n+l 



dx 



(C.l) 



where c(p) is introduced in (|67|) and 

h(r 



lim Im 



sin arcsin(r4e* a ) 

T 

T 4 



r > 1. 



(C.2) 
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Since 



+0+ I TTt 



r-n+1 



lim Im i — f ^ X ~\ ^ - dx 

+0+ 7T /i X n+i 



lim Im i — f — — '} da; 

+0+ 1 7T h x n+1 



1 f 00 Im{A(x + ei) - 1} , 
= lim — / — dx 

e^0+ 7T Ji X n+1 

then for n > 1, we can rewrite (|C.lj) in the equivalent form 



— Re < 



(1-p) 



2 roo 



9piT 



h(r) 



1 I 1 + c(p)re 3 



dr> + Bmi Tim «* + «)-!} 



f.n+1 



dx (C.3) 



and the function /i(-) in (|C.2|) can be expressed in a simpler way as 



hir) = ^ 



1+Vl-r 2 



1 - V 1 - r a 



r > 1. 



(C.4) 



Although we will not make use of this in the sequel, we note that in this representation, the 
function smoothly interpolates A n for non-integral values of n. 

We start by bounding the absolute value of the first term in the RHS of (|C.3|) . From QC.4JI . 
it follows immediately that h(r) is positive and monotonic increasing in r for r > 1, and that it 
is upper bounded by 2~ (which is the limit of h(r) as r — » 00). We therefore get from the 
non-negativity of c(p) in (|67|) that 
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We note that (|C5|) decreases by at most a factor 
result later in (|U.12|) . 
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(C.5) 



when we increase n to n + 1. We use this 
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Regarding the second integral in the RHS of (|C.3|) . we note that for x > 1 

t(( x -l)\^^ 

'l-V \ 2 1-v)1 



lim — Im{ A(x + ei) — l\ 

e ^ + 7T I J 



7T V 3p 



(x-l) 



where 



= Im "I e 4 sin (— arcsin ( ue 1 



,u > 0. 



Now, we rely on the inequality (which is shown in Fig. EJ) 

1 X6 

> /(«) = -u - — u 5 , u>0. 
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Figure 9: A plot which verifies graphically the inequality in ()C.8 



It follows from dTTfil) . JTTTTfc and (ITTsl) that 
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Equality (|C.9|) follows from the identity 

(x-l) Q , T(a + 1) T(n - a) 

dx = — — — - — i-r , -1 < a < n 
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where T(-) designates the complete Gamma function [the integral in the LHS of (|G.10j) can be 
calculated by the substitution x = which transforms it to L (1 — u) n ~ a ~ 1 u a du, and then the 
latter integral is by definition equal to B(a + l,n — a) where £>(•, •) designates the Beta function; 
this function is related to the complete Gamma function by the equality B(m,ri) = r(m+n) •] ' 
Since T(x + 1) = xF(x), and T (iA = y^r, then we obtain from (|C.10|) that 

dx r (l) £(n-j) = ^r(n-i) 
x n+l r(n+l) 2T(n+l) 



30 r(|) r(n-|) (j - § • § - ^ • V^) r(n-i) 105 y^r(n-i) 

x r(n+l) r(n+l) ' 16 r(n + i) 

which confirms the equality in (|C.9|) . 



We note that (|G.9|) (which forms a lower bound on the second term in the RHS of (|C.3)l ) 
decreases by at most a factor 1 — 2 {n+i) uniformly ° ver P when we increase n to n + 1 (it can be 
shown by using the recursive equation T(x + 1) = xT(x) for the complete Gamma function). 

From (|G.3|) . ()G.5|) and (|G.9j) and since c(-) in (|6T)) is a monotonic decreasing and non-negative 
function over the interval (0, 1), it now follows that if for a specific n* G N and for all p G [0,p*] 

r(n*-|) 315p 2 !>*-!) ^ 2^3(1 + ^))-^ 



2(l-p)V5Fr(n* + l) 16(l-p)70Fr(n* + l) " pl^n* - 1 



— r > 1 (C.12) 

2(n* + l) " + c (p*) 



then \ n {p) > for all n > n* and p G [0,P* 



In the following, we consider the positivity of the sequence {A n (p)} for p G [0,0.95]. From the 
above bounds, it follows that we can prove the positivity of this sequence in some band p G [0,p*] 
by checking the positivity of only a finite number of terms in this sequence (we note that this 
number grows dramatically for values of p* which are very close to 1). Assume we pick p* = 0.95. 
By explicitly evaluating (|G.11|) . we see that we need n* > 7957. From condition (|G.12j) . we get 
n* > 4144, so a valid choice for the fulfillment of both conditions is n* = 7957. Based on (jG.ll|) 
and (|C.12|) . we conclude that X n (p) > for all n> n* and p G [0,p*]. For n < n* and all p G [0, 1), 
we will verify the positivity of the coefficients {X n (p)} by alternatively showing that for n < n* , 
the polynomials P n { ) in <|72|) are positive in the interval [0, 1]. This equivalence follows from (|71j) . 
First we can observe from ()75(l that since P n (0) and -P n (l) are positive for all n G N, then the 
polynomials P n (-) are positive in the interval [0, 1] if and only if they have no zeros inside this 
interval. Hence, based on Lemma |31 (see Appendix B), one can verify the positivity of P n {p) for 
n < n* and p G [0, 1] by simply verifying the positivity of P n *(-) in the interval [0, 1]. To complete 
our proof, we proceed as follows. We write 

2(n-l) 

Pn(p)= £ b^(p- Po y (c.i3) 

where for convenience we choose Po = \ (this will be readily clarified). Based on (|69|) where 
Pn(p) = EST" «i"V, it follows that 

t w = rfW =E (A (» Vo -.. (c . 14) 

3>i V 7 
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Therefore, from the recursive equation (|70j) for {a- n } and from (|C.14|) . it follows that all coefficients 
{fr^} are rational and can be calculated from the coefficients {al } which are defined recursively 
in (|70[) and are rational as well. Using an infinite precision package, those coefficients can be 

(n) 

computed exactly. By explicit computation, we verify that all the coefficients are strictly 
positive for n = n* and < i < 2{n* — 1), and therefore it follows from (|C.13|) that P n *{-) is 
positive (and strictly increasing) in the interval [po>l]- F° r P 6 [0,Po]; ° ne can verify from the 
conditions in (|C.11|) and (|C.12|) that X n+ i(p) and P n (p) are positive for n > 57 and p G [0,po]- 
Combining these results, we conclude that P n *(-) is positive in the interval [0,1]. This therefore 
concludes the proof that X n (p) is positive for all n £ N and p £ [0,0.95]. 

Though not proving the positivity of A n (-) over the whole interval [0,1), we note that the 
uniform convergence of the plots which are depicted in Fig. (see Appendix B) and (j71|) strongly 
supports our conjecture about the positivity of A n (-) over this interval. 
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Figure 1: The Tanner graph of IRA codes. 

Figure 2: The function A(-) in (|15j) . as a function of the erasure probability (p) of a BEC. 

Figure 3: The exact and approximated values of A n +i (n = 1, . . . , 1000) for the check-regular IRA 
ensemble in Theorem |2 The approximation is based on the asymptotic expression in (JTHJ). The 
plots refer to a BEC whose erasure probability is p = 0.2 (left plot) and p = 0.8 (right plot). 

Figure 4: BER and WER for random rate 0.925 codes from the bit-regular IRA ensemble in 
Theorem^with q = 3 and the regular systematic IRA ensemble with d.d. X(x) = x 2 and p(x) = x 36 . 
The curves are shown for N = 8000, 64000, and 500000. 

Figure 5: BER and WER for random rate ^ codes from the check-regular IRA ensemble in 
Theorem 0and the check-regular LDPC ensemble [E] for N = 8192,65536, and 524288. 

Figure 6: The branch cuts of the function \(x) in (|15jh and the contour of integration in Eq. ()B.1|) . 

Figure 7: Plots of isijjiM) f or the polynomials P n (-) which are introduced in Eq. ()72l) . In the 
limit where n — > oo, these curves converge uniformly on [0, 1] to the function f(x) = 21n(l + 2x). 

Figure 8: The branch cuts of the function X(x) in (|15|) . the original contour of integration of radius 
r, r < 1, and the modified contour with radius R tending to infinity. 

Figure 9: A plot which verifies graphically the inequality in ()C.8|) . 
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